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ABSTRACT 


This thesis examines the use of random sampling as a 
basis for taking extrapolated deductions in the payment of 
Navy maintenance service contracts. Motivated by increasing 
inspection requirements, NAVFAC has authorized limited use 
of this quality assurance tool. The experience of two 
activities which implemented extrapolated deductions in 
their family housing maintenance contracts is the focus of 
this study. Methods employed are evaluated along with an 
analysis of payment history, contractor performance and 
inspection requirements. The statistical basis used in the 
field tests along with an alternate proposal is analyzed. 
Various issues are identified which should be addressed 
before general guidance is promulgated. Major 
recommendations deal with sample adjustment, computer 


support, inspection methods and the need for oversight. 
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i ~INTRODUGTEION 


A. BACKGROUND 

Random Sampling for Extrapolated Deductions (RSED) 
involves making an inference concerning a population based 
on a Simple random sample. The results of that sample are 
then used as a basis for equitable payment across the entire 
contracted service. As a Simple example, suppose that a 
Sample inspection of services revealed 10% not performed. 
The contracting officer would then estimate that 10% of all 
services were not performed, and reduce the contract price 
by 10%. While payment deductions for observed poor perforn- 
ance or nonperformance by service contractors is standard 
practice, the extrapolation of deductions based on random 
sampling to the general population is currently prohibited 
by NAVFAC regulation. This policy has been under review in 
recent years. 

Interest in this technique is a result of the increasing 
role that service contracts play in accomplishing the base 
maintenance function and the subsequent strain placed on the 
inspection force. As part of a move toward formulating Navy 
policy, two activities were granted authority to use 
extrapolated deductions in their Family Housing Maintenance 


Contracts. 


Bee OBJECTIVE 

The objective of this research is to evaluate the field 
use of random sampling for extrapolated deductions in Navy 
maintenance service contracts. This is being done to 
identify issues which should be addressed before general 
authority is granted and guidance promulgated. In addition, 
proposed NAVFAC statistical methodology is evaluated in 


light of information obtained in the course of this study. 


C. SCOPE AND ASSUMPTIONS 

This study focuses on the procedures used to implement 
the field testing of random sampling for extrapolated 
deductions in Navy maintenance service contracts. Air Force 
use of this contracting technique is addressed to provide a 
background and contrast to the NAVFAC effort. Experiences 
one other service contracting organizations with 
extrapolating deductions either within or outside the 
Federal Government was not pursued. It is assumed that the 
reader is familiar with the concept of simple random 


sampling. 


D. RESEARCH METHODOLOGY 

A literature search was conducted to review regulations, 
reports, industrial literature, and policy guidance applica- 
ble to statistical quality assurance and service 


contracting. Contact documentation and history was obtained 


from the following organizations involved in the initial use 
of extrapolating deductions: 

1. Public Works Center, San Francisco, Ca; and 

2. Public Works Center, Great Lakes, Il. 

Additionally, interviews were conducted either in person 

or by telephone at the following organizations: 

1. Public Works Center, San Francisco, Ca; 

2. Public Works Center, Great Lakes, Il; 


3. Western Division, Naval Facilities Engineering 
Command, San Bruno, Ca; 


4. Public Works Center, San Diego, Ca; 


5. Public Works Department, Naval Postgraduate School, 
Monterey, Ca; 


6. Public Works Department, Marine Corps Air Station, El 
TOL One Car, 


7.  Vandenburg Air Force Base, Lompoc, Ca. 


E. THESIS ORGANIZATION 

This chapter identified the current status of NAVFAC 
regarding the extrapolation of deductions in service con- 
tracting and presents the objectives and methodologies of 
this study. Chapter II puts the study into perspective with 
a discussion of the maintenance service contracting environ- 
ment. Chapter III traces the historical development of 
statistical quality assurance techniques and their applica- 
tion to service contracting. Chapter IV examines random 
sampling, NAVFAC policy concerning RSED, and two statistical 


approaches that have received consideration as a basis for 
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extrapolating deductions in NAVFAC’ service contracts. 
Chapter V presents findings and conclusions resulting from 
an evaluation Ce two activities implementation Gt 
extrapolated deductions in their family housing maintenance 
eontracts. Finally, Chapter VI lists recommendations for 
the general use of extrapolating deductions and makes 


Suggestions for future research. 


el 


If. MAINTENANCE SERVICE CONTRACTING ENVIRONMENT 


A. BACKGROUND 

A Maintenance Service Contract (MSC), commonly referred 
to as simply a service contract, calls for a contractor's 
time and effort rather than a finished product. Service 
contracts are funded by the organization having budgeted 
responsibility for the services acquired. Examples of 
service contracts include refuse collection, guard services, 
grounds maintenance, pest control, and family housing 
maintenance. A MSC is normally entered into for a one year 
period with an option to extend for two additional one year 
periods. However, extensions to the contract cannot exceed 
the fifth anniversary of contract award. [Ref. 1] 

While not a critical distinction for purposes of this 
paper, a MSC is one category of a Facility Support Contract. 
The other category is a Maintenance Construction Contract. 
Due to the nonrepetitive nature of construction and the 
higher degree of importance associated with each unit of 
work, they are not normally amenable to statistical quality 
assurance and will not be a focus of this study. 

Facility Support Contracts are designed to accomplish 
the maintenance and repair of real property facilities. 
They can best be thought of as an extension of the base 


public works force, and in fact replace the in-house forces 
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where a decision has been made to contract out a function 
previously performed by Public Works. 

Service contracts are almost exclusively of the fixed 
price type. It is the belief of NAVFAC that fixed price 
contracts force contractors to perform efficiently. Where 
uncertainty of workload precludes a firm fixed price, 
Indefinite Quantity contracts are awarded on the basis of 
firm unit prices for an estimated workload. This study 
encountered the extrapolation of deductions being applied 
only to firm fixed price contracts, however the technique is 


also applicable to firm unit prices. [Ref. 1] 


B. TREND TOWARDS SERVICE CONTRACTING 
Service contracting in the public sector’ gained 
attention in 1954 when, in his first budget message to 
Congress, President Eisenhower said: ([Ref. 2] 
This budget marks the beginning of a movement to shift to 
.- « »« private enterprise Federal activities which can be 
more efficiently carried on that way. 
This was followed by Bureau of the Budget Bulletin 55-4 
which implemented the stated policy as follows: [Ref. 3] 
It is the general policy of the administration that the 
Federal Government will not start or carry on any commer- 
Cial activity to provide a service or product for its own 
use if such service or product can be produced from 
private enterprise through ordinary business’ channels. 
Exceptions to this policy shall be made by the head of the 
agency only where it is clearly demonstrated in each case 
that it is not in the public interest to procure such 
products or services from private enterprise. 
Today's policy comes from OMB Circular A-76, issued in 


1966 and further revised in 1967, 1979 and 1983. Ge 
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basically states that whenever a commercial activity is 
Federally operated, the cost of operation should be compared 
to the cost of contracting to determine who should do the 
work. A commercial activity (CA) is defined as a product or 
service obtainable from a commercial source. [Ref. 4] 
Although it has been federal policy to contract out 
commercial functions since the 1950s, the program was 
largely ignored until the 1979 revision of A-76 [Ref. 5]. A 
key element of that version required government forces to 


submit a "firm bid" based on a clear Statement of Work 


(SOW). Contractors would also bid on the requirements 
contained in the SOW. Thus the open market would determine 
the contracting decision. Award would be to the lowest 


responsible bidder if under the government estimate by 10%. 
To ensure that the government estimate was properly 
prepared, it was subject to approval by the Naval Audit 
Service [Ref. 6]. 

Thus, the fact that DODI 4100.33 required a complete 
review of all CA functions and a compliance system was in 
place set the stage for a number of functions formally 
performed by Public Works to be contracted out. 

Data compiled in January 1982 showed that 60% of all 
functions reviewed shifted to contract, at an overall cost 
savings of 19%. Of all NAVFAC functions studied, the areas 
of transportation, groundS maintenance, building mainte- 


nance, and janitorial service accounted for 82% of the total 
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Preis” 7]. The next few years will undoubtedly see conver- 
sions occur at an ever increasing rate as the great majority 


of reviews are completed. 


C. PERFORMANCE 

Service contracts, while not directly mission related, 
are usually of great concern to the activity commanding 
officer. Visible lack of performance reflects directly on 
his ability to effectively maintain the shore establishment 
entrusted to his care. The result is acute command interest 
in performance, an interest which is reflected throughout 
the contracting organization. 

Acceptable performance in service contracts is often 
difficult to achieve. While a consensus of opinion did not 
emerge during personal interviews, frequently cited reasons 
included a lack of inspector resources, inadequate specifi- 
cations, and a lack of enforcement techniques to motivate 


the contractor. 


D. QUALITY ASSURANCE 

Quality Assurance (QA) is the procedure by which the 
government verifies that it is receiving the services for 
which it contracted and paid for. Quality Assurance 
Evaluators (QAE) are responsible to the Service Contract 
Manager (SCM) fom actiad monitoring OF Soneraccor 
performance. The QA program is contained in the QA Plan. 


it details how the QAE will monitor all aspects of a 
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contractor's performance, although it is not part of the 


actual contract. 


Several surveillance methods may be used as part of the 


QA Plan. Those selected are based on the unique considera- 
tions of an activity. Techniques currently in use are: 
[Ref. 8] 

a) One Hundred Percent Inspection--Appropriate for infre- 


b) 


Cc) 


d) 


e) 


quently performed services or those of great impor- 
tance, it is expensive and time consuming. 


Planned Sampling--Designed to inspect some part of the 
contract requirement. It cannot be used to make an 
overall judgment concerning contract performance, but 
is useful where performance in a particular area or 
location is of interest. 


Unscheduled Inspection--An impromptu check. 


Validated Complaints--Originate with the customer. 
The method is successful to the extent that customers 
are aware of and can distinguish good performance. 
While not recommended as a stand alone procedure, it 
can be used to make payment deductions and require 
rework. 


Random Sampling--A statistically based procedure which 
attempts to incorporate the best features of 100% 
inspection and planned sampling. By inspecting a 
sample representative of the entire population, it 
allows a judgment to be made about overall performance 


with a reduced level of inspector resources. A 
properly constructed random sample may also be the 
basis for extrapolating deductions. However, the 


extrapolation of deductions is currently not allowed 
by NAVFAC regulation and is the focus of this study. 


E. ENFORCING PERFORMANCE 


If the government accepts anything less than 100% of 


contract bid requirements, the contracting officer must take 


some appropriate action. Not to do so could indicate that 
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the government has overstated its requirements resulting in 
an illegal contract. 

Other than the routine identification of contractor 
deficiencies, negative motivators form the basis of NAVFAC 
approved techniques to enforce performance. Cont Lace 
actions other than payment reduction are all designed to 
threaten and support eventual default for failure to 
Perrtorm. The effectiveness of these techniques are 
questionable because of the difficulty in sustaining 
termination of a borderline contractor. In addition, the 
prospect of extended litigation and interruption in services 
must be considered. Conversations with contracting officers 
indicate that "living" with less than acceptable performance 
until contract expiration is the best of a bad situation. 
It is reasonable to assume that a savvy contractor is aware 
of the contracting officer's dilemma. 

The most effective performance motivator currently 
permitted is the withholding of payment for unsatisfactorily 
performed services. Withholding payment assumes a strong 
profit motive on the part of a capable contractor, a "fair" 
contract under which reasonable profits are possible, anda 
schedule of deductions which results in reduced profits when 
exercised. Other possible contractor motivations such as 
growth or maintenance of the workforce do not appear valid 
in service contracting. Contractors generally do not staff 


for a job until contract award, and a ready supply of 
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unskilled or semiskilled laborers makes adjustment of the 
workforce an easy thing to do. 

Current NAVFAC regulations allow deductions for work not 
acceptably performed, unless the contractor is given the 
opportunity to re-perform the work and does so satisfactor- 
aeliyee If feasible, rework is preferable to deductions 
because of the emphasis on performance. When taken, deduc- 
tions are in accordance with individual bid items if an 
Indefinite Quantity contract, or an agreed-upon schedule of 
deductions for Firm Fixed Price contracts. However, this 
contract action is limited by the fact that appeals boards 
and courts uniformly hold "penalty" provisions unenforce- 
able. Section 9-302.1 of the Navy Contracting Manual 
applies: [Ref. 1] 

No contract may provide for penalties for non-performance, 
as such are legally unenforceable. However, a contract 
may provide for liquidated damages, which is an advance 
agreement between parties to a contract as the damages one 
party will suffer if the other fails to perform... Note 
that if a contractor does not perform work, it would be 
improper to withhold payment for that work AND also charge 
him the full cost of having the work done by others, for 
that would result in the Navy having work done at the 
expense of others. 

Liquidated damages of 10% are assessed whether or not 
the work is subsequently performed by the contractor to 
compensate the government for increased administrative 
costs. Where government forces re-perform the work, a 20% 


liquidated damage charge is levied along with the full cost 


of the government effort. [Ref. 1] 
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Current NAVFAC regulations allow deductions for observed 
defects only. The extrapolation of deductions based on 
random sampling 1S prohibited [Ref. 8]. tis ) policy 
resulted from the lack of an acceptable statistical tech- 
nique and questions as to the legality of extrapolated 
deductions. However, NAVFAC has recognized the potential of 
extrapolating deductions to further motivate the contractor, 
or at least to better match payment for services received. 
Therefore they have initiated efforts in recent years to 
define a methodology under which extrapolation might be 
authorized for general use. iimatpporst Of this .effort, 
NAVFAC has authorized trial tests of extrapolating deduc- 
tions in family housing maintenance contracts at PWC San 


Francisco and PWC Great Lakes. 
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III. HISTORICAL OVERVIEW OF STATISTICAL 
QUALITY CONTROL 
A. EARLY EVOLUTION 

Since man began to manufacture and produce products, 
there has been concern over quality of output. As far back 
as the Middle Ages, the medieval guilds insisted on 
apprenticeships and required that those seeking to become 
master craftsmen offer evidence of their ability. Such 
rules were in part a form of quality control. In more 
modern times, factory inspection and research, activities of 
professional societies, and various regulations have sought 
for years to assure a quality of output. Quality control 
has thus had a long history. 

On the other hand, statistical quality control is rela- 
tively new. The science of statistics itself goes back only 
about 300 years. The theory of probability originated in 
1654 when Pascal, a French philosopher and mathematician, 
teamed up with Pierre Fermat to develop the new science. 
Early applications were made in astronomy and physics, but 
it was not until the 1920's that statistical theory began to 
be applied effectively to quality control. It should be 
remembered though, that even before the 1920's, industry was 
learning to do things more scientifically. Techniques such 


as the GANTT chart, under principles developed by Taylor, 


20 


Gilbreth and others were a part of the movement called 
"Scientific management." 

Walter Shewhart, of the Bell Telephone Laboratories, was 
among the first to apply statistical methods to the problem 
@eeeguality control [Ref. 1]. Two other Bell System men, 
H.F. Dodge and H.G. Romig, took the leadership in developing 
the application of statistical theory to sampling 
inspection. The culmination of their work being the now 
well-known Dodge-Romig Sampling Inspection Tables. An 
immense amount of work went into the preparation of these 
tables, for at that time hand calculators were the only 
available aid to speedy calculations. These tables were 
published in 1941 [Ref. 2]. The work of Shewart, Dodge, and 
Romig constitutes much of what today comprises the theory of 
statistical quality control. 

When this new inspection plan was proposed there were 
many skeptics. Therefore, in the initial trials, after 
performing sampling inspection, every lot was then inspected 
100% to check conformity. The checks showed that sampling 
did give good results and with much less time and money used 
for inspection. After its initial application in the 
central offices of Western Electric Company, the new proce- 
dures were tried in the Hawthorne manufacturing plant in 
Chicago, and proved a big success. 

The rate of adoption of these newer methods were slow in 


the United States. Professor Freeman, who was promoting 
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statistical quality control at Massachusetts Institute of 
Technology, ascribed this sluggish response of the early 
years to: (1) the conviction of American production 
engineers that their principal function was to improve 
quality control through advances in technology, (2) a 
prevailing attitude that the laws of chance had no place in 
a scientific field, and (3) the scarcity of adequately 
trained industrial statisticians. In 1937, probably not 
more than a dozen single enterprises in American mass- 
production industry had introduced the new techniques into 


their ordinary operations. [Ref. 3] 


B. INFLUENCE OF WORLD WAR II 

World War II overcame the initial coolness of American 
industry toward statistical quality control. The armed 
services began to enter the market as large consumers of 
American output and, as such, had an increasing influence on 
quality standards. A group of engineers from Bell Telephone 
were brought to Washington to develop a sampling inspection 
program for Army Ordnance. This step was made necessary by 
the bottlenecks occurring at inspection stations during full 
scale production. 

The group from Bell faced a sampling problem of unprece- 
dented size. It was necessary that all vendors be treated 
fairly; it was necessary to get enormous quantities of ord- 
nance material quickly; and it was necessary that arms and 


ammunition be reliable and safe. The "acceptable quality 
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control" system that was developed encompassed the concept 
of protecting the consumer from getting unacceptably defec- 
tive material, and encouraging the producer in the use of 
process quality control. This was done by varying the 
quantity and severity of acceptance inspections in direct 
relation to the importance of the characteristics inspected, 
and in inverse relation to the quality level indicated by 
those inspections [REf. 4]. The same men also carried out 
an extensive training program to acquaint government person- 
nel with the new procedures and tables. Many terms used 
extensively today, most notably "producers risk" and 
"consumers risk," had their origins in this original effort. 

Wartime research in statistical quality control was 
carried out primarily by the Statistical Research Group, 
Columbia University, Applied Mathematics Panel, Office of 
Scientific Research and Development. This research group 
was organized in July 1942, and continued in existence until 
September 1945. Members of the staff were drawn from 
universities around the country. It advised and assisted 
the Army and Navy on statistical aspects of problems arising 
in their activities. Included in its contributions to sta- 
tistical quality control was the preparation of the Navy 
manual on sampling inspection by attributes [REf. 5]. 
Inspection by attributes requires that the inspected item be 


categorized either as totally acceptable or not acceptable. 
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Professor A. Wald, also of the Statistical Research 
Group, developed the idea of sequential sampling. In 
sequential sampling, the sample size is not determined in 
advance. The sequential procedure is to inspect one item at 
a time. The inspection is continued until enough cumulative 
evidence has been gathered to either accept or reject a lot. 
A pre-assigned risk of making incorrect decisions controls 
the ultimate same sample size. This procedure allows very 
good lots to be accepted quickly and bad lots to be rejected 
quickly. It is only with lots of doubtful quality that 
extensive sampling is required. This analysis was deemed so 
important by the U.S. Government that it withheld publica- 


tion of Wald's original paper until June 1945. [Ref. 5] 


C. MILITARY STANDARDS (MIL-STD-105) 
1. Development 

In the post war period there were two systems of 
sampling plans and tables, Army tables and Navy tables. The 
procedures and tables were alike in general plan and struc- 
ture but differed in a number of respects. In 1949, a basic 
Military Standard, MIL-STD-105, was developed using material 
from both service tables. Military Standard 105D, the 
current version, is the outcome of a study by an American- 
British-Canadian working group that sought to derive a 
common standard for the three countries. MIL-STD-105D was 
issued by the U.S. Government in 1963. In 1971, it was 


adopted by the American National Standards Institute as ANSI 
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Z1.4 and in 1973, it was adopted by the International 
Organization for Standardization as ISO/DIS-2859. [Ref. 6] 
2. Acceptance Sampling 

Acceptance sampling is the method of evaluating a 
group of units, drawn from a production lot, in order to 
determine the acceptability of the whole lot. It has been 
widely accepted as the preferred method of acceptance con- 
trol. MIL-STD-105D provides tables which facilitate the 
implementation of an Acceptance Sampling by Attributes plan. 
The focus of the plan is an acceptable quality level (AQL). 
An AQL represents that quality level which, for purposes of 
Sampling inspection, can be considered satisfactory as a 
process average. FRe fers |] 

In addition to an initial decision on an AQL, it is 
also necessary in applying MIL-STD-105D to decide on an 
"inspection level." This determines the relationship 
between the lot size and the required sample size. Three 
inspection levels are provided for general use with guidance 
for switching between them. 

For a particular sample size and AQL, the tables 
will specify a reject number. When the number of defects in 
a sample is equal to or greater than the reject number, the 
entire lot is judged to be unacceptable. The sample size, 
reject number, and AQL define an "operating characteristic" 
curve for a particular plan. This curve depicts the 


relationship between the actual defect rate submitted for 
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inspection and the corresponding probability of acceptance 
of such lots by the sampling plan. It is based on the 
binomial distribution. 
The nature of acceptance sampling is explained by 
Siegmund Halpern: {[Ref. 15] 
Inasmuch as only a portion of the whole lot is inspected, 
there is always a risk that the quality of the sample will 
not reflect the quality of the lot. As a result, produc- 
tion lots may, at times, undeservably be rejected and 
faulty lots accepted. The former will be of interest to 
the producer. He wants to minimize the risk of having 
good lots’ rejected. This risk is identified as the 
producer's risk. The risk of accepting bad lots in a 
sampling scheme is, of course, the primary concern of the 
consumer. This risk is called consumer's risk. The 
method by which both the producer's risk and the 
consumer's risk are quantified statistically is by 
constructing an operating characteristic curve. 
Acheson Duncan also comments: fRef. 9] 
It is to be emphasized that the purpose of acceptance 
sampling is to determine a course of action, not to 
estimate lot quality. Acceptance sampling specifies 
procedures that, if applied to a series of lots, will give 
a specified risk of accepting lots of a given quality. 
Appendix A illustrates how the sampling plans 
contained in MIL-STD-105D are used. A typical operating 


characteristic curve is also presented. 
3. Application to Air Force Service Contracting 
Random sampling as a quality assurance tool for 
monitoring contractor services was introduced to the 
military by the Air Force in 1979 with AFR 400-28. MIL-STD- 
105D provided the basis for the random sampling procedures 


contained in the regulation. [Ref. 16] 
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Originally designed to determine the acceptance or 
rejection of manufactured lots, MIL-STD-105D was easily 
adapted to the service contracting environment. Specific 
instances of work could be judged "defective" or acceptable, 
aS required in attribute sampling, the same as a manufac- 
tured part. Lot size was normally determined by the number 
of services provided in a month, which corresponded to a pay 
period. A major difference between the use of acceptance 
sampling for manufactured items and evaluating service con- 
tract performance is that a "lot" of services could not be 
totally rejected if found unacceptable, since it had already 
occurred. Therefore, contract actions based on the results 
of acceptance sampling must recognize that some satisfactory 
work has taken place for which the contractor is due 
payment. 

AFR 400-28 contains tables derived from MIL-STD-105D 
that show acceptance and rejection numbers corresponding to 
specified acceptable quality levels and sample sizes. The 
acceptance number is the maximum number of defects allowed 
before a sample is considered unsatisfactory. When the 
number of defects in the sample exceeds the acceptance 
number, contractor performance is deemed unsatisfactory and 
contract payments can be reduced for the service provided 
during the sample period. 

Allowable AQLs are limited to those which appear in 


MIL-STD-105D. AFR 400-28 calls for the establishment of 
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reasonable AQLS as part of the original CA solicitation. 
Thus contractors bid on performance requirements of less 
than 100% for various services. Contractors are also 
provided with the activities QA plan as an enclosure to CA 
solicitations. This is done to inform prospective bidders 
as to how the contract will be monitored and to help them 
set up their own quality control program. It is Wiec- 
however, a part of the contract. 

The extrapolation of deductions is a key feature of 
AFR 400-28. When the random sampling method of surveillance 
is used and the service is rejected, contractor payments may 
be reduced in the same proportion for the entire lot as the 
percentage of defects found in the sample. For example, if 
a rejected sample disclosed that 15% of an inspected service 
was defective, contract payments for the service may be 
reduced by 15%. However if the number of defects found do 
not equal or exceed the reject number, the services are 
judged satisfactory and no deductions are taken. 

Numerous problems have been experienced at the 
activity level in implementing AFR 400-28. An audit con- 
ducted by the Air Force Audit Agency in 1985 to evaluate 
quality assurance of base-level service contracts reported 
the following findings: (Ref. 17] 


a) for seven of twenty contracts reviewed, QAES made 
errors in the use of random number tables; 


b) for fifteen of twenty contracts reviewed. QAEs did 


not correctly use random sampling techniques due to 
the complexity of AFR 400-28; 
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c) for ten of twenty contracts reviewed. QAEs did not 
properly apply accept/reject numbers due to the com- 
plexity of the AFR 400-28 acceptance tables; and 

ad) sampling plans were not an effective means of deter- 
mining contractor's performance as the plans used fre- 
quently required the acceptance of services even 
though the percentage of defects occurring in the 
sample was approximately twice the defect rate per- 
mitted in the contract. 

The Air Force is now investigating a simplified 
random sampling process based on Indifference Quality 
Levels. In such a plan, the government and the contractor 
equally share in the probability of acceptance or rejection 
at a specified quality level [Ref. 17]. 

4. Navy Rejection of MIL-STD-105D 

NAVFAC service contracts are written requiring 100% 
performance. Thus the Navy is more interested ina statis- 
tical methodology which will provide information as to 
actual contract performance rather than the accept/reject 
hypothesis testing of MIL-STD-105D. 

A random sampling procedure was designed by NAVFAC 
in 1979 to determine the level of contractor performance. 
Published in the MO-327 [Ref. 8], which is the Navy equiva- 
lent of AFR 400-28, it was not intended to be a statistical 
basis for extrapolating deductions. The sampling plan 
contained in MO-327 made use of an AQL that was not releasa- 
ble to the contractor. This AQL differed from the one used 


in MIL-STD-105D in that it was defined to be "tolerable" 


performance rather than acceptable. 
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The new contract surveillance technique was not 
widely used. Conversations with contract administrators 
indicated that this method of random sampling was hard to 
understand, difficult to administer, and provided question- 
able results upon which to base contract actions. Adding to 
that the fact that extrapolated deductions were not 
permitted, there remained little incentive for its use over 
the more understandable planned-sampling technique. 

In 1982, NAVFAC determined that the extrapolation of 
deductions was appropriate as long as there was a degree of 
confidence that deductions made were reasonable and not 
construed as penalties. Their approach is studied in the 


following chapter. 


30 


IV. RANDOM SAMPLING FOR EXTRAPOLATED DEDUCTIONS 


A. INTRODUCTION 

Random Sampling for Extrapolated Deductions’ (RSED) 
involves making an inference concerning a population based 
on a Simple random sample. The results of that sample are 
then used as a basis for equitable payment across the entire 
contracted service. 

There are three reasons why the concept of extrapolating 
deductions is supported by NAVFAC at this time: (1) it is 
believed to provide extra motivation for the contractor 
through the potential of increased deductions, (2) extrapo- 
lation results in a better match of payment to services 
received, and (3) the ability to extrapolate deductions 
makes the use of random sampling more attractive as a 
quality assurance tool. Given the current’ service 
contracting environment, the ability to obtain acceptable 
evaluation of contractor performance with reduced levels of 
inspector resources would appear to be the predominant 
issue. 

The application of extrapolated deductions must be based 
on a statistically valid random sample if a protest is to be 
upheld. Most activities have had no experience with random 
sampling and the guidance available is limited in value. 


The next section looks at the considerations involved in 


a 


employing random sampling in a- service contracting 


environment. 


B. RANDOM SAMPLING 
1. Characteristics 

A properly constructed random sample must ensure 
that every possible sample of a given size has an equal 
chance of being selected. Additionally, the process being 
inspected must be homogeneous and produce defects in a 
random manner. If this is not the case, a selection bias 
may result which causes the attributes of the sample to be 
different from the population. [Ref. 18] 


This theoretical ideal, approachable in manufactured 


lots, cannot realistically be attained in service 
contracting. Some rooms are going to be harder to clean 
than others. A bus service will be more likely to lag 


behind schedule during certain hours of the day. If grounds 
maintenance is provided at two different bases, a SCM would 
not be happy with a sample which required 95% of the 
inspections at one base. Thus, a sampling plan must be 
devised which will "reasonably" represent the population of 
services. A general rule would be to adopt a selection 
method that will give every member of the population an 
equal chance of being selected. In general, it is best to 


use random numbers. These will hold up in court [Ref. 9]. 
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2. Advantages 

The use of random sampling as a quality assurance 
tool has inherent advantages over other inspection tech- 
niques. The most obvious is that it provides a cost effec- 
tive way of estimating the true level of contractor 
performance. The fact that it may now be used as a basis 
for extrapolating deductions makes it all the more attrac- 
tive. Some other benefits of random sampling offered by 
contract administrators are: 


a) Requires less supervision of inspectors to make sure 
they are using their time effectively; 


b) 1s not susceptible to the predictability which may be 
involved in planned sampling; 


c) allows inspection of services to be transferred easily 
from one inspector to another as no previous knowledge 
of the contract is required. 

NAVFAC has recommended that random sampling be 
applied to service contracts possessing the following 


characteristics: [{Ref. 8] 


a) Composed of a large homogeneous population of work 
units, 


b) individual work items not of a critical nature, 
c) population size estimable, 
a) inspector travel time not excessive. 
3. Problems in Practical Use 
Although there would seem to be numerous advantages 
involved with the use of random sampling, even without 
extrapolated deductions, its use seems to be very limited 


within the NAVFAC community. Conversations with various 
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contract administrators indicate that a lack of familiarity 
with random sampling procedures, the complexity of writing 
amenable specifications, and the confusing guidance in the 
MO-327 are at the heart of their reluctance. 

Probably the major impediment to instituting random 
sampling plans, with or without extrapolated deductions, is 
the lack of random sampling guide specifications for 
individual contract types. Contract administrators are 
either not willing or able to get bogged down in the details 
of generating performance specifications and quality 
assurance plans which properly incorporate random sampling. 
For that reason, a great many services amenable to random 
sampling are being inspected in a judgmental manner with 
poor performance defined as an intolerable level of customer 
complaints. 

Janitorial services would seem to be a prime candi- 
date for random sampling, yet interviews with contract 
administrators revealed confusion and frustration where 
attempts to write a suitable plan were made. This confusion 
is easily understood. Questions exist over what the sampled 
unit should be. Should it be individual rooms, groups of 
rooms, buildings, or the specific acts of emptying ashtrays 
and cleaning windows? Also, how is a representative sample 
taken when conditions vary? Suppose that a BOQ building is 


randomly selected for inspection. Can the percent of rooms 
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found to be nonconforming be used as an indicator of overall 
performance across the various buildings on base? 

The design of random sampling plans is beyond the 
scope of this paper. It is addressed because of its 
discovery as an issue in the course of the research. Due to 
the complexities involved with designing random sampling 
plans and the lack of specific guidance, I would not be 
Surprised if the majority of random sampling plans which in 
effect are of poor statistical validity. Contractors have 
had no reason to challenge these procedures either because 
they did not understand them, or because they were not being 
used as a basis for deductions other than observed deficien- 
cies. It should be anticipated that the extrapolation of 
deductions will cause many of these sampling plans to come 
under intense scrutiny. 

Another impediment to the use of random sampling has 
been the difficulty of generating random numbers and apply- 
ing it to the population. Appendix B contains instructions 
from the M0O-327, which provides the guidance for random 
sampling and tables for random number generation. Sample 
generation by the manual method can be seen to be time 
consuming and difficult to apply objectively. The tendency 
would be for users to become frustrated with unusable 
numbers and just resort to making up numbers on their own. 
While NAVFAC recognizes the shortcomings of the random 


number table and now advocates the use of computer programs 
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for generating samples, they still recommend the use of the 
tables where a computer and program are not available. 
PRef.> 197) I believe that a computer based system is a 
necessary part of establishing a viable program with 


complete command support. 


C. EXTRAPOLATING DEDUCTIONS 
1. Background 

NAVFAC did not adapt the statistical methods of MIL- 
STED-105D for surveillance of NAVFAC service contracts 
because it called for acceptance of work at levels below 
100%. Additionally, it allowed large deviations in the 
observed sample above the AQL before it was rejected. 
Reservations about the legality of extrapolated deductions 
Was also expressed. [Ref. 20] 

In 1983, NAVFAC adopted the thinking that extrapo- 
lated deductions would be an effective contractor motivator. 
It was also thought that it would provide an additional 
incentive for activities faced with scarce inspector 
resources and increasing levels of contracted services to 
adopt random sampling. The technique would be implemented 
when a methodology was available to take deductions fairly, 
with a high degree of confidence that they would not take 
the form of punishment. Dr. Douglas Montgomery of the 
Georgia Institute of Technology was hired as a consultant to 


develop a statistical basis and procedure for extracting 
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these deductions [Ref. 20}. Working with NAVFAC, a 
methodology was ready for field use in 1984. 

The ultimate form that extrapolated deductions will 
take is still in doubt. NAVFAC is currently working on a 
new statistical basis which is more in line with its 
contracting philosophy. The elements of that methodology 
will also be looked at in this chapter. 

2. Legal Considerations 

The legal basis for extrapolating deductions is 

derived from the Federal Acquisition Regulations: [Ref. 21] 


If any of the services do not conform with contract 
requirements, the Government may require the contractor to 


perform the services again. .. . When the defects in 
services cannot be corrected by reperformance, the Govern- 
ment may .. . (2) reduce the contract price to reflect 


the reduced value of the services performed. 
A key concept is that the reduction in price must 
correspond to the cost of the services. A case that applied 


this ruling was Environmental Aseptic Services Administra- 


tion and Larson Building Care Inc. [{Ref. 22]. This case 
involved an Air Force janitorial contract. The protesters 


complaint was that sampled services, which in this case was 
"cleaned rooms," subsumed several required tasks (e.g., 
aseptic floor, furniture, fixtures, drapes, and trash). The 
Performance Requirements Summary provided that if a task 
fails, the room failed for the day. In other words, if the 
contractor failed to clean the ashtrays, he would suffer the 
same deduction as though he had failed to perform all 


required tasks in the room. Thus the extrapolated deduction 
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greatly exceeded the value of the one task not performed. 
The GAO ruled in the case that the deduction was tantamount 
to a penalty, and its inclusion in the Invitation for Bids 
unnecessarily raises the governments cost and has an adverse 
effect on competition. 

Closely tied to the issue of reasonable cost deduc- 
tions is the treatment of liquidated damages. The Federal 
Acquisition Regulations state the following policy: [Ref. 
23] 

Liquidated damages clauses should be used when both (1) 
the time of delivery or performance is such an important 
factor . . . that the Government may reasonably expect to 


suffer damage if the delivery or performance is delinquent 
and (2) the extent or amount of such damage would be 


difficult or impossible to ascertain or prove. .. . The 
rate of liquidated damages used must be reasonable and 
considered on a case-by-case basis. .. . Liquidated 


damages fixed without any reference to probable actual 
damages may be held to be a penalty and therefore 
unenforceable. 

The Navy Contracting Manual, P-68, allows liquidated 
damages to be assessed o FSCS at the rate of 10% of the 
price of the service whether or not it is subsequently re- 
performed. The extrapolation of deductions has caused some 
confusion in this area. Do liquidated damages apply to the 
estimated portion of unperformed work? 

Original NAVFAC guidance indicated that the extrapo- 
lation of liquidated damages was proper [Ref. 24]. Later 
guidance called for liquidated damages to be applied only to 


items re-performed, which would prohibit their extrapolation 


(Ref. 19]. More recent discussions with NAVFAC indicate 
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that the policy will be to take liquidated damages for 
observed deficiencies, whether or not it is subsequently re- 
performed. 

The key question to be asked concerns the reason for 
liquidated damages. If it is to compensate the government 
for increased administrative costs associated with re- 
inspection, then it iS appropriate to apply liguidated 
damages only in instances where the item is re-performed. 
However, if the basis of the liquidated damages is to 
compensate the government for the interruption of normal 
activities associated with dirty chalkboards and unemptied 
dumpsters, then re-performance iS not a consideration and 
the damages remain. Also, the damages associated with the 
estimated portion of nonperformance would be just as real as 
the inspector observed deficiencies. The fact that NAVFAC 
now advocates that liquidated damages be applied for 
observed deficiencies would indicate the latter as a basis. 
Therefore, the extrapolation of liquidated damages would 
appear defensible in court. The greater amount of deduc- 
tions assessed would also act as an enhanced motivating 
force. 

Another important issue from a legal standpoint is 
the implications of not obtaining a sample by procedure. 
For example, suppose an inspector fails to obtain the sample 
required by procedure. Does that invalidate the sample for 


extrapolation purposes? From the standpoint of violating 
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the procedure itself, apparently not. The GAO has ruled 
that when regulations set out instructions clearly for the 
benefit of the government, the agency's failure to comply 
with it does not provide a basis for protest. [Ref. 25] 

As far as the sample size itself, the government's 
failure to select an adequate sample has been the basis for 
upholding protests [Ref. 25]. However, that case involved a 
sampling scheme which was part of the contract. NAVFAC has 
no plans to make their quality assurance program a contract 
requirement. Even in the case where a sampling scheme has 
been included as a part of the specifications, the Armed 
Service Board of Contract Appeals has ruled that a deviation 
in sampling procedure is allowable where "no appreciable 
difference in results occurs" [Ref. 26]. I agree with the 
NAVFAC policy not to include quality assurance procedures in 
the’ -contrace- The government must reserve the right to 
remain flexible in this area. At the same time, contrac- 
tually specified quality assurance would open up contract 


administrators to numerous protests. 


D. STATISTICAL BASIS FOR FIELD TESTING 
1. Ineeoduceien 
The statistical method devised by D.r Montgomery for 
use in the NAVFAC field tests are easily incorporated into 
tables amenable for field use. These tables require no 
knowledge of statistics for application. However, a lack of 


appreciation for the relationships which form the basis of 
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the tables means that the potential for misusing them 
exists. Lacking specific guidance, activities may unneces- 
sarily increase their administrative burden in cases of 
mismatched samples and populations. It should be emphasized 
that the motivation behind extrapolated deductions is to 
better match pay to services using minimal inspector 
resources. This section looks at the statistical basis of 
the field tests and discusses the various components upon 
which the tables are based. 
2. Use of the Tables 

Tables approved by NAVFAC for use in monitoring 
service contracts and taking extrapolated deductions were 
Gistributed to Engineering Field Divisions and Public Works 
Centers in 1984. Guidance in use of the tables (see sample 
tables in Appendix C) was provided by NAVFAC as follows: 
(Ref. 24] 


In order to use the tables, the user must first establish 
the "Critical Performance Level" (CPL) which is defined as 
the level of performance by the contractor that the 
government considers to be adequate and is expressed as a 
percentage. The CPL is established during the specifica- 
tion writing phase and should be entered into the Perform- 
ance Requirements Summary. For a given CPL and population 
size, the tables give the user a sample size and a 
Critical Number of Defects (CND). The CND is the number 
of defects in the sample that would signal poor contractor 
performance and is the threshold for considering taking a 
deduction based on a statistically estimated number of 
defects. When the number of defects observed in the 
sample equals or exceeds the CND, the number of defects in 
the population is estimated by multiplying the population 
by the percentage of defects found in the sample. When 
the number of defects observed in the sample is less than 
the CND, deductions will be taken for observed defects 
only. . . . The Normal Surveillance Table will usually be 
used for a new contract. The Tight Surveillance Table 
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should be used if observed defects exceed two-thirds of 
the CND for two consecutive months. The Reduced Surveil- 
lance Table should be used if the observed defects are 
less than one-third of the CND for two consecutive months. 
Normally when Reduced Surveillance Tables are used, deduc- 
tions will be taken for observed defects only. 

Use of the term CPL (equal to 100-AQL) was NAVFAC's 
way of stating that 100% performance was being contracted 
for, but that a somewhat lower level would be tolerated 
before adverse contract action would be initiated. The 
inclusion of the CPL in the contract was NAVFACsS first 
concession in writing that less than 100% performance was 
acceptable. This change in policy was made necessary by the 
fact that the CPL was the key factor in taking extrapolated 
deductions, and in the course of explaining deductions to 
the contractor it would have become known anyway. 


Opinion iS mixed whether or not the CPL being part 


of the bid package would lead to lower bids by contractors 


planning to under-perform. The incentive to do so would 
seem to be there. For example, if the CPl was 90%, a 


contractor could cut his bid by 10% and plan to only perform 
90% of the work. If he could maintain his actual perforn- 
ance at 90%, he could avoid extrapolation 95% of the time. 
Although he would have to rework observed deficiencies or 
suffer the specific deductions, these would only be a small 
percentage of the actual deficiencies which existed. The 
result of all this would be a borderline contractor with 
high administrative costs. This possibility points out a 


weakness of including the CPL in the contract under this 
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type of system. Limiting the CPL to the QA plan would allow 
it to be changed at a later date if it became necessary. 

It should be noted that the method approved is very 
Similar to that used by the Air Force. For a given perform- 
ance requirement and population size, a CND (reject number) 
1s established at which time extrapolation can be applied. 
The major procedural difference is that below the CND the 
Navy would still deduct for observed defects, while the Air 
Force did not. This difference was due to the Navy's basic 
requirement being 100% performance. 

3. Mathematical Basis of Field Method 

The mathematical basis of the tables used for the 
field test is the normal approximation of the hypergeometric 
probability distribution. The sample sizes shown in the 


tables were calculated using the relationship of equation 


(1): 
en 
2 
1 
n = [—-—_" + © (1) 
me Grea CP Is) 
where: 
n = required sample size 
CPL = critical performance level 
N = estimated population size 
Z = normal one-sided z-statistic 


W = width of the confidence interval = (1-CPL)/2 
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The z-statistic is dependent on the confidence level 
desired. For normal inspection the confidence level is 95% 
and the corresponding z-statistic is 1.645. This means that 
the resulting sample will ensure that the observed defect 
rate is within the specified confidence interval 95% of the 
time. Tightened inspection equals 99% confidence, while 
reduced equals 90%. 

4. Effect of CPL 

The confidence interval (W) can be seen to be a 
function of the CPL. It is perhaps easier to think of it as 
one-half the allowable defect rate. Thus the greater the 
CPL, the more narrowly defined is the confidence interval, 
and the greater is the sample size required. This relation- 
ship may be undesirable due to the lack of specific guidance 
in setting CPLs. Contract administrators have interpreted 
the CPL to be a level of acceptable performance from an 
average contractor. Thus easily performed services should 
rate a high CPL regardless of the importance of the service. 
Because the confidence interval then becomes such a small 
number, which is subsequently squared, it has an overriding 
effect on the sample size. From the normal table (see 
Appendix C) it can be seen that if the population estimate 
is 1000 units of work, a shift in the CPL from 90% to 95% 
would result in the required sample size being doubled. 

If the confidence interval is going to vary 


inversely with the CPL, a distinction should be made between 
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"Critical" services and those which happen to rate a high 
CPL due to the ease with which they can be performed. 
Perhaps the latter category could be inspected at a reduced 
level of confidence. This would be consistent with the 
efficient use of inspector resources. 

The use of the CPL term in equation (1) is only a 
best estimate of actual performance, which is what is 
actually mathematically required to make the equation valid. 
If the actual performance is less, the actual confidence 
level achieved will be less. NAVFACS rationale for 
accepting lower confidence levels when Conerdctor 
performance is below the CPL is that greater attention will 
be focused on the contractor's performance by various other 
methods. A plot of this relationship for a 95% confidence 
level (normal surveillance) and CPLs of 90% and 97% is shown 
in Figure 4.1, a population of 300 was assumed. It can be 
seen that achieved confidence at realistic performance 


levels still gives the benefit of the doubt to the contrac- 


EOr . Therefore, whether or not other actions are antici- 
pated, very poor performance does not invalidate 
extrapolation. 


5. Effect of Sample Size 


In Section C-2 of this chapter, the implications of 
not taking the appropriate sample size was discussed. 
Figure 4.2 shows a plot of the relationship between sample 


size and confidence level for populations of 100 and 500, 
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Figure 4.1 Confidence Level as a Function of Performance 
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Figure 4.2 Confidence Level as a Function of Sample Size 


normal sampling and a CPL of 90% are assumed. Where actual 
samples fall below the required sample, achieved confidence 
gradually decreases. This is especially true where large 
samples are involved. For example, if the sample taken for 
a population of 500 was only half the required 82, achieved 
confidence would still be in excess of 86%. It can be 
concluded that failure to reach a designated sample is 
probably not going to appreciably affect the results. 
Therefore guidance on this matter should be promulgated to 
avoid the situation of activities invalidating their samples 
when there is no need to do so. 
6. Effect of Population Estimate 

Similar to the situation of not obtaining the proper 
Sample size is the almost monthly problem of not properly 
estimating the population size. Analysis of the relation- 
ship between confidence level and population size shows that 
this situation is of almost no consequence. Take the case 
where the estimated population is 500 units and the desired 
confidence level is 95% (CPL = 90%, sample size = 82). If 
the actual population turns out to be 400, the resulting 
confidence level will be 95.5%. At an actual population of 
600, the original sample of 82 will still give a confidence 
level of 94.7%! In fact, whenever the original sample size 
is greater than about 30, the achieved confidence level 
should be sufficient to support extrapolated deductions 


using the original CND. Again, there is no guidance on this 
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matter and activities may try to unnecessarily match sample 
sizes to ultimate populations at the expense of manpower and 
possibly statistical validity. It would appear to be the 
case that in all situations where random sampling is appro- 
priate, the original sample should be sufficient to support 
extrapolated deductions or any other contract action. 
7. Operating Characteristics 

The CND is calculated by determining the defect rate 
which exceeds the CPL plus the confidence interval at the 
specified confidence level. Equation (2) illustrates this 
relationship: 


CND = on[(CPL + Z PL) (Pacrh (1 >) (2) 


The application of equation (2) 1s equivalent to a 
standard hypothesis test. The statement being made at a 
normal level of inspection is that if the CND is reached, we 
are 95% confident that the actual defect rate is equal to or 
greater than the allowable defect rate. 

Figure 4.3 contains an operating characteristic 
curve for various CPLs. It can be seen that while this 
methodology is biased toward the contractor, it does exhibit 
higher levels of producer risk than MIL-STD-105D. A good 
point of comparison is an actual performance rate of 85% and 
a CPL of 90% (AQL = 10%). The NAVFAC statistical basis will 


accept this level of performance just over 50% of the time, 
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Figure 4.2 Characteristics of Normal Sampling. sample=50 
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while the Air Force would accept it 85% of the time using 


MIL-STD-105D (see Appendix A for MIL-STD-105D). 


E. PROPOSED STATISTICAL BASIS FOR FUTURE USE 

A problem with the statistical basis used in the field 
tests, aS viewed by NAVFAC, is that it did not allow for 
extrapolation until contractor performance was below the 
CPL. This implied that performance above the CPL was 
acceptable. Yet NAVFAC is adamant that service contracts be 
written for 100% performance. Thus, a new method for taking 
extrapolated deductions is being developed by NAVFAC. It is 
planned that when ready for field use, this method will 
replace the method used in the field tests. This section 
describes the characteristics of the new method as it 
currently exists. 

The proposed statistical basis for taking extrapolated 
deductions in the future is more in line with NAVFAC's 
policy of contracting for 100% performance and taking 
deductions where evidence of nonperformance exists. Code 
named RSED V3.0, this methodology seeks to determine the 
contractor's actual level of performance. Deductions are 
then extrapolated without regard to a CPL. The concept of a 
CPL is utilized to the extent that if a contractor's 
performance does not meet the CPL, as determined by the ODR, 
contract actions in addition to payment deductions are 
indicated. These CPLs are to be included in the contracts 


Performance Requirements Summary. [Ref. 19] 
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RSED V3.0 has not been finalized for field use. Prelim- 
inary indications are that it will incorporate one basic 
sampling level, although others may be added. The defect 
rate determined by the sample is adjusted downward by a 
specified "delta" depending on the observed defect rate to 
determine a level appropriate for deductions. The use of 
the delta ensures that the deduction rate will be less than 
or equal to the contractor's actual defect rate 75% of the 
time on a monthly basis, and approximately 90% of the time 
over the annual contract term. The higher confidence level 
associated with the contract term is felt by NAVFAC to be 
the minimum necessary for which to base stronger contract 
actions such as termination or not renewing options. 
Appendix D contains the sample size and delta tables along 
with a sample payment analysis reproduced from NAVFAC 
documentation. [Ref. 19] 

Like the model used for field testing, RSED V3.0 uses 
the normal approximation of the hypergeometric distribution 
to determine sample sizes and confidence intervals. The 
relationship used for calculating sample sizes is identical 
to the field model except that different assumptions are 


made. Equation (3) illustrates: 
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where: 


n = required sample size 


W = .01 (a fixed value of 1% regardless of CPL) 

P = .95 (a fixed value of anticipated performance, 
not CPL) 

Z = .674498 (normal z-statistic for 75% certainty) 


N = estimated population size 


Equation (3) can be rewritten as, 


n = [.00462749 + 1)71 (4) 
N 

The resulting sample size can be seen to be independent 
of the CPL. It will yield an observed defect rate within 1% 
of the actual defect rate 75% of the time, provided the 
actual performance is close to the estimate of 95%. At the 
end of the month, the observed defect rate is used as a 
better estimate of actual performance to calculate a new 
confidence interval. This new confidence interval is found 
by solving Equation’ (1) jforwe wi The new value for "Ww" 


becomes the "delta" adjustment. Equation (5) illustrates: 


_ Poste 20) 
DELTA = @ aoe xz CS) 
where: 
P = observed defect rate (ODR) 


N 
l 


1.28159 (normal z-statistic for 95% certainty) 


52 


= 
| 


= 12,000 


Fie 32 


The "delta" is calculated with fixed values of "N" and 
'"n" so that one table which is a function of the observed 
defect rate may be utilized. This simplification results in 
deltas approximately one percent less than would have been 
obtained using actual sample sizes and populations. It also 
precludes the necessity of having a table for each possible 
Sample size and population! The delta in conjunction with 
the ODR can now be used to estimate the true level of 
contractor performance, and to determine a deduction rate to 
be used for extrapolation. The following statements help 
summarize the results of this methodology using an ODR of 
10% as an example: "We are 75% confident that the true 
defect rate is greater than the deduction rate of 8.62% 
(ODR-DELTA),: or for evaluating performance, "We are 75% 
confident that the true performance level of the contractor 


is less than 91.38% (100% - ODR + DELTA)." 


F. COMPARISON OF THE TWO STATISTICAL BASES 
1. Inspection Requirements 
The proposed statistical basis will require greater 
sample sizes to be inspected than the field version. This 
is due to the fact that RSED V3.0 seeks to ascertain actual 
contractor performance within a tolerance of one percent. 


Because extrapolation will be applied at all levels of 
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performance under RSED V3.0, this small confidence interval 
is deemed necessary by NAVFAC. Since the field version is 
designed only to determine whether actual performance is 
less than a "tolerable" level, it can be done with a smaller 
Sample at a higher level of confidence. 

Another difference between the methods is’ the 
greater flexibility allowed with the field version. Depend- 
ing on inspector resources and contractor performance, SCMs 
can direct inspection at three confidence levels with large 
differences in inspection requirements. In addition, 
different CPLsS can be assigned to further adjust inspection 
requirements within the different levels. However, the 
large influence on inspection requirements exerted by the 
CPL may cause some contract administrators to specify a 
performance level lower than actually desired. Under normal 
levels of inspection, specifying a 95% CPL over a 90% CPL 
results in almost doubling the sample size. The potential 
thus exists for contract performance requirements to become 
a function of inspector resources. 

Increased flexibility could be built into RSED V3.0. 
This would have to be done by allowing different confidence 
intervals, since the confidence level of 75% would seem to 
be approaching a minimum. Specifying larger confidence 
intervals would not change the ODR, which will be the basis 
for contract actions other than extrapolation and remains 


the best estimate of contractor performance. Also, the fact 
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that straight line deductions exist in the Air Force where 
contract specifications have not been met, implies that 
NAVFAC is being unnecessarily conservative. 

2. Effect On Contractor Performance 

It is reasonable to assume that the contractor will 
be motivated closer to 100% performance under RSED V3.0. 
However, most contract administrators feel that 100% 
performance is unrealistic for most services, and expressed 
the opinion that public works forces rarely performed at the 
now specified CPL. The extrapolation of deductions at 
performance levels which have been previously considered 
satisfactory represent a source of increased risk to the 
contractor which may be expected to result in higher bid 
prices. 

Bid prices may also be higher under RSED V3.0 for a 
more positive reason. The feature of extrapolating at all 
defect levels means that there is a much closer match 
between payment and services received. Contractors will not 
have any incentive to underbid the contract with the goal of 
performing at the CPL and avoiding extrapolation. Thus RSED 
V3.0 might result in more bids where the contractor is able 
to make a fair profit at the required performance level. 
Whether or not this results in higher overall bids depends 
on the profit levels now being enjoyed by’ service 


contractors. 
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There seems to be a greater potential for an adver- 
Sarial relationship to develop between the contract 
administrators and contractor under RSED V3.0. This is 
because every observed deficiency would have ramifications 
beyond its own merits. Under the field method, this wasn't 
the case until the CND was reached. At that point there was 
little room for sympathy because the contractor was perform- 
ing at levels well below specified levels. Because the 
quality of services is subject to some inspector judgment, 
borderline cases under RSED V3.0 may be dismissed as 
inspectors carry the increased burden of identifying 
discrepancies which will be the basis for extrapolation. 

3. Deduction Level 

There are too many variables involved to try and 
quantify the anticipated differences in deductions which 
would be obtained using the two statistical methods. at 
would be fair to say that the potential for higher deduc- 
tions exists under RSED V3.0. The dichotomy remains that 
when either system is functioning as intended, the deduc- 
tions will be at a minimun. 

A major difference between the two methods is the 
treatment of discrepancies outside the random sample (e.g., 
customer complaints, planned sampling). When deductions are 
extrapolated, it is not correct to also deduct for deficien- 
cies found outside the sample, as these would be 


theoretically included in the extrapolated results. Since 
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RSED V3.0 extrapolates from the start, it is never 
permissible to deduct for a customer complaint. The field 
method would allow this deduction if the CND had not been 
reached. An interesting result is that in services which 
are customer oriented and a strong customer complaint pro- 
gram exists, extrapolated deductions under RSED V3.0 could 
easily be less than what would have been obtained by 
deducting for known deficiencies. This does not appear to 
be a situation that the contractor can take advantage of. 
Even if the services were not amenable to re-work, contract 
actions other than extrapolation could be resorted to. 
4. Producer/Consumer Risk 

Both the field version and RSED V3.0 are biased 
towards the contractor. The field version allows actual 
defect rates to be approximately 1.5 times the CPL before an 
even chance of rejection occurs and straight line extrapola- 
tion is applied. Performance levels under RSED V3.0 are 
closely determined, but then a "delta" subtracted from the 
ODR assures that the contractor will be overpaid in the long 
alr. 

A normal approach to comparing the producer bias 
inherent in any statistical techniques is the operating 
characteristic curve. However, Since there is no reject 
number (CND) involved with RSED V3.0, a curve for it cannot 
be constructed. It is safe to say that in either method, 


the contractor is treated more than fairly. A theoretical 
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even treatment would be to apply the ODR as a deduction rate 
at any performance level. However, NAVFAC is not willing to 
live with the idea that in any one month, the contractor has 
an even chance of being overpaid or underpaid, least not at 
reasonable inspection levels (confidence intervals). 
5. Ease of Use 

Both methods appear workable and of comparable 
aifficulty to implement and administer. However, under RSED 
V3.0 required sample sizes are specified for much narrower 
population ranges. This increases the potential for 
mismatched samples with actual populations. If clear and 
Simple guidance isn't promulgated which defines actions to 
be taken when procedures aren't strictly followed, the 


administrative burden could become excessive. 
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A. INTRODUCTION 
1. Purpose 

The purpose of this survey is to identify general 
issues which should be addressed before the extrapolation of 
deductions is approved for general use. Although both of 
the field tests involved family housing maintenance con- 
tracts, the focus of this study remains oriented toward 
general issues. Preferred methods for applying extrapolated 
deductions to any specific contract type are outside the 
scope of this work. 

eee background 

The two activities involved in the field test are 
Public Works Center (PWC) Great Lakes and PWC San Francisco. 
Although the housing maintenance requirements of each 
activity are similar, many differences exist regarding both 
the circumstances of the contracting effort and the approach 
taken to employ extrapolated deductions. Key differences 
include CPL requirements, performance weighting, and the 
method for classifying service work for bid and inspection 
purposes. 

Both maintenance contracts were initially written as 
the basis for a CA review. In the case of San Francisco, 


the contract ran for two years before approval to use 
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extrapolated deductions was granted in July, 1985. Prior to 
that time, random sampling had been conducted using the 
procedures of MO-327. 

At Great Lakes, extrapolated deductions was a part 
of the contract from its inception in 1984. NAVFAC provided 
assistance in developing contract specifications, but the 
QAP and related payment documents were generated in-house 
from scratch. The methodology for taking deductions was 
approved at the NAVFAC level before extrapolation was 
allowed. 

3. Data Collection 

The collection of quantitative data was made diffi- 
cult due to the great amount of paperwork which is generated 
over the course of administrating a service contract. I was 
limited to using payment summary sheets because of the 
fragmentary nature with which relevant information is 
located within that paperwork. These summaries were not 
directly comparable, so some information obtainable at one 
activity could not be collected at another. One example was 
an inability to determine the percent of rework accomplished 
at Great Lakes. This prevented a deduction comparison with 
RSED V3.0 at that activity, which was done with the PWC San 
Francisco data. 

Data collection was focused in both cases on perform- 
ance, inspection and payment history of service work. 


Service work is normally customer generated and requires 
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less than 16 hours to complete. It constitutes the largest 
component of work performed under a housing maintenance 
contract, approximately half. Random sampling was applied 
to service work for the purpose of extrapolating deductions 
at both activities. Other major portions of housing 
maintenance includes change-of-occupancy work and preventive 
maintenance. Data on these categories of work was not 
collected because the change-of-occupancy work is 100% 
inspected and the preventive maintenance data was not easily 
summarized. 

Data collected at PWC San Francisco represented a 
contract period of 11 months. Data collected at Great Lakes 
extended over a 24 month period. In addition to a collec- 
tion of historical data, interviews were conducted with 


contract administrators at both activities. 


B. CONTRACT REQUIREMENTS 
1. Treatment of Service Calls at Great Lakes 
Service calls at Great Lakes were classified as 
either emergency, urgent, or routine. The implication for 
the contractor was that each required different response 
times. The definitions of each category are given below 
along with the required response time. 
EMERGENCY CALLS--These are problems which pose an immedi- 
ate danger to the occupants or have the potential to cause 
further property damage (e.g., power lines down, overflow- 


ing toilet). They require contractor response within 30 
minutes on a 24 hour basis. 
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URGENT CALLS--These are problems which do not pose an 
immediate threat to occupants or property, but would soon 
inconvenience and affect the well-being of the occupants 
(e.g., refrigerator not working, toilet backed up). The 
contractor is required to respond within two hours of the 
cally: 

ROUTINE CALLS--These are problems which cannot be classi- 
fied as either an emergency or urgent situation (e.g., 
tile repair, binding doors). The contractor is required 
to respond within nine days and to complete the work 
within 14 days. 

The performance of service calls was broken down 
into four performance indicators: responsiveness, quality 
of work, proper classification, and proper accounting. Each 
one of the performance indicators was assigned a weighting 
representative of the "worth" of that portion of the job. 
Performance weights must total 100%. When performance 
indicators are weighted, it means that accomplishment of any 
individual performance indicator earns a percent of the 
service price equal to the weighting. Thus, weighting has a 
direct bearing on the level of deductions to be assessed for 
failure to perform. For example, the performance weighting 
for emergency response is 45%. Suppose that a contractor 
responded to an emergency call one hour late. Regardless of 
the fact that quality work was done and the job was properly 
documented, 45% of the price would be subject to deduction. 

2. Treatment of Service Calls at San Francisco 
Service calls at San Francisco were also categorized 


as emergency, urgent and routine. However the required 


response times differed as follows: 
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EMERGENCY CALLS--two hours during work hours, four after; 
URGENT CALLS--twenty four hours; 


ROUTINE CALLS--five days response, fourteen days 
completion. 


San Francisco also used the same four performance 
indicators as Great Lakes, but did not assign weights to 
them. In effect, that meant that each one was weighted 
100%. A deficiency in response time, quality, or documenta- 
tion would result in a total deduction being taken. 

Another difference between the activities was that 
San Francisco combined its service calls into three other 
categories for the purpose of bidding and inspection. These 
were interior, exterior and appliances. Thus, exterior work 


might consist of any of the three service call types. 


C. QUALITY ASSURANCE PLANS 
ieee nit roauct 10n 
Quality Assurance Plans (QAP) are long, complex 
documents which attempt to provide the activity with a 
structured approach for inspecting all aspects of a contrac- 
tor's performance. It is beyond the scope of this paper to 
attempt to convey the contents of either activity's QAP. My 
intent is to report only those findings which have general 
applicability to RSED. 
2. General Comments 
A major difference between the two plans was that 


PWC Great Lakes wrote theirs initially to incorporate RESED. 
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San Francisco's plan was based on the M0O-327 version of 
random sampling, which did not allow for extrapolation. The 
result was that the QAP in San Francisco had little 
resemblance to the quality assurance program which 
eventually evolved. The fact that PWC San Francisco was 
able to implement a workable program after some trial and 
error, reinforces the wisdom of separating the QAP from the 
formal contract. 
3. Critical Performance Levels 

The CPLs contained in the QAP were not part of the 
bid information of either contract. NAVFAC's policy then, 
which has just recently changed, is that the CPL was an in- 
house tool only. However, while unknown to the contractor 
during the bidding process, the contractor soon became 
familiar with the CPL as it formed the basis of payment 
analysis and performance judgments. 

The CPL assigned to service contracts in San 
Francisco was 90%, while Great Lakes enforced a CPL of 97%. 
San Francisco felt that a CPL in excess of 90% was too harsh 
and unrealistic. The opinion at Great Lakes was that 97% 
performance was realistic and had been achieved by the 
departing public works force. The SCM at Great Lakes also 
felt that a 97% CPL was high enough to prevent planned 
noncompliance by the contractor. A possibility he thought 


plausible at a CPL level of 90%. 
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The actual weighted average performance or service 


calls achieved at PWC San Francisco was 92.0%, as compared 


to 98.1% at PWC Great Lakes. While contractor performance 
is subject to many variables, ine hud thag inspector 
aggressiveness, the close parallel between performance 


differences and CPL differences suggests a relationship. 
4. Performance Weighting 

The failure of PWC San Francisco to weight the 
different aspects of service call performance for payment 
purposes would probably have made extrapolation unsupporta- 
ble in court. This is similar to the Air Force janitorial 
contract [REf. 2]] where a room could be declared unsatis- 
factory and payment denied because an ashtray was not 
emptied. 

Not weighting performance factors also makes bidding 
more difficult and transfers greater risk to the contractor. 
This may have been a factor in PWC San Francisco paying a 
weighted average of $55.00 per service call, while Great 
Lakes paid only $33.00. 

There was not enough data to support conclusively 
whether or not weighting influenced contractor performance 
at Great Lakes (see performance weights in Figure 5.1). 
However, in emergency and urgent calls where responsiveness 
and quality were closely weighted, mean performance was 
within one half of a percent of each other. In the case of 


routine calls, where quality carried twice the weight of 


65 


SAyHeET 2PS4UQ DQMd 2e GupzyHtam aduewsuoy4uay JZUadDVaY T°SG aunbry 


BO ° a0°* O>* oe 4 S199 ANILNOY 
BO" ao ° OS ° Ov ° 911V9 INSIYN 
BO° ao" St" Sb * 91189 AINSIYAWS 


INTLNNOI9IV NOILVOISISS¥V19 ALIWNO ASNOAS3Y 


66 


responsiveness, quality performance exceeded responsiveness 


ved . 3%. 


D. INSPECTION REQUIREMENTS 
iP ie Eoauct lon 

This section looks briefly at the use of inspector 
resources at each activity. Any random sampling program 
approved for extrapolated deductions must permit efficient 
use of inspectors, with minimal administrative burden, if it 
is to receive command support. Although many of the inspec- 
tion procedures referred to are in the QAP, all inspection 
related information is presented in this section for 
eilarity . 

2. Sampling Levels 

Figures 5.2 and 5.3 show the monthly level of 
inspection employed at each activity over the period studied 
along with the population range for each of the service 
categories. It should be noted that due to the lower CPL, 
inspection levels at San Francisco were far below that of 
Great lakes. In fact, although the number of calls at San 
Francisco are higher, they inspected less than half the 
calls inspected at Great Lakes. This relationship would 
suggest too great an incentive for contract administrators 
to sacrifice performance requirements for the sake of saving 
inspector resources. 

These figures also show the potential savings 


possible by combining the services into one group for 
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Figure S.2 Inspection Data at PWC Great Lakes 
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Figure 3.35 Inspection Data at PWC San Francisco 
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inspection purposes. If the population of service calls at 
San Francisco were combined into one group of 1433 calls, 
the required sample size would only be 91!. Whether or not 
this could reasonably be considered a homogeneous population 
is subject to debate. Differences in frequency or cost is 
no greater than the variation which occurs within each 
service call category. The economy inherent in large 
populations argues for combining work items where possible. 

A look at the population ranges in Figures 5.2 and 
5.3 illustrates the difficulty in estimating an appropriate 
sample size. Another benefit to increasing the population 
is the greater stability involved. A sample size of 91 
would apply to all populations in the range of 1300-1500. 

3. Surveillance Level Switching 

Both QAPs provided conditions under which surveil- 
lance levels could be changed. At PWC San Francisco, the 
option was not exercised for two reasons: (1) contract 
administrators felt that reduced inspection levels would not 
have been adequate, and (2) tight surveillance was beyond 
the capacity of the inspector force. 

Surveillance levels were switched as deemed appro- 
priate by the SCM at PWC Great Lakes. Although the guidance 
in the QAP was not strictly followed, the switching tech- 
nique was effectively utilized. Even within the routine 
service work category, different surveillance levels were 


used for a period of time. Response time was under tight 
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inspection, while quality of work was inspected under 
reduced surveillance. It should be remembered that the 
incentive for going to reduced inspection was much greater 
at Great Lakes. Even under reduced inspection, at a 97% CPL 
level, greater inspection was required than normal inspec- 
tion at San Francisco. 

The switching of inspection levels does present a 
problem in balancing the workload. The SCM at Great Lakes 
claimed that he was staffed for a normal inspection load. 
When tight inspection was implemented, as it was for the 
first year under routine response, it could be covered only 
through the use of overtime and the shifting of inspectors 
from other assignments. In my view, the only thing gained 
by tightened inspection is the ability to extrapolate the 
Same percentages at a higher confidence level and a few more 
observed deficiencies. It would seem more efficient to stay 
at normal inspection for extrapolation purposes and use 
planned inspections where it was necessary to prod the 
contractor a little further. Planned inspections on a 
limited basis can usually be done so that the incidence of 
deficiencies is enhanced. Also, these extra planned 
inspections would not be a firm requirement and would thus 
allow extra flexibility. While deductions resulting from 
planned inspections are allowed only if extrapolation does 


not take place, the rework requirement is still enforceable. 
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4. Sample Generation 

Both activities made use of a computer program to 
generate random samples. At the beginning of the month, the 
computer would generate the random numbers and these would 
be matched to service call chits as they came in. 

The better program was in use at Great Lakes in that 
it did not require the use of tables (see tables in Appendix 
C). Required input consisted of a population estimate, CPL, 
surveillance level and lowest sample number. At San 
Francisco only the population estimate and sample size was 
required for input, but determining the appropriate sample 
size required using the tables. Aside from the enhanced 
possibility of making a mistake, the stratification of the 
tables normally meant that required samples were slightly 
greater than if determined directly by formula. im 
addition, the San Francisco program always generated numbers 
starting from one, which meant that chits needed to be 
renumbered every month. 

5. Sample Adjustment 

The question of sample adjustment arises when the 
actual population of services in a month requires a 
aifferent sample than the one obtained as a result of the 
prior estimate of the population. At PWC San Francisco, 
this possibility was not addressed in the QAP. Lacking 
clear guidance, command procedure has been to evaluate the 


sample as if the population estimate had been correct. This 


ak 


is a Simplifying procedure based on conserving inspector 
resources. In evaluating the sampling history, this 
experience has been justified. Due to the relatively large 
Samples involved, the lowest being appliances at a mean of 
64 inspections per month, little change in confidence levels 
would have been gained by doing additional sampling. fThere 
was not time where an adjustment could have resulted in 
extrapolation. 

The procedure written for PWC Great Lakes was more 
sophisticated, yet was unnecessary. At the end of the 
month, the QAE adjusted the sample size using the computer 
program. By giving it the same "seed" number, it 
automatically generated a new sample which either adds a few 
numbers to the previous sample or deletes a few. The QAE 
would then either inspect the additional samples or invali- 
date some already inspected for purposes of extrapolation. 
It is never desirable to invalidate a sample, as this only 
reduces the confidence level of the results. 

As practiced, the SCM decides whether or not to do 
sample adjustment based on the possibility of extrapolation. 
Where the number of defects is close to the CND, he will 
authorize a sample adjustment. Even this procedure is not 
always prudent. Suppose the estimated population for 
emergency calls is 175. At a CPl of 97% and normal surveil- 
lance, this would call for a sample size of 117 and a CND of 


six. If the population turned out to be 200, it would call 


a2 


for a sample size of 127 and a CND of seven. Assuming the 
Original number of defects found was six, the inspector 
would be sent out to sample ten more jobs in search of one 
more defect. However, had extrapolation been assessed on 
the basis of the original sample, the unsatisfactory deter- 
mination would have been made at a confidence level of 93%. 
Certainly this is good enough to ascertain unsatisfactory 
performance. Assuming that the original estimate had been 
made in good faith, this extrapolation would no doubt have 
been upheld in court. It would appear that guidance is 
necessary in this area, so that activities can make the most 


intelligent use of their resources. 


E. PAYMENT ANALYSIS 
Paebncroduction 
Payment analysis at both activities was based on 
general NAVFAC guidance issued early in 1984 [Ref. 24]. 
Bach activity modified the guidance slightly to make it more 
applicable to payment for service call work. This section 
evaluates the methodology of taking extrapolated deductions 
at each activity. The deductions which resulted are then 
analyzed in light of the performance realized. 
2. Payment Methodology 
Figure 5.4 is an example of payment analysis for 
emergency service calls at Great Lakes. The procedure 
followed at San Francisco is illustrated by a reproduction 


of the payment calculation for exterior maintenance in 
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Setober, 1985. This calculation is shown in Figure 5.5. 
The obvious difference between the two is that performance 
weighting requires separate analysis for each performance 
indicator at Great Lakes. 

The wording of line "I" instructions (see Figures 
5.4 and 5.5) was found to have caused problems at both 
activities. Line "I", as used at Great Lakes, is identical 
to NAVFAC guidance. However, the confusing instructions 
caused an inspector to not extrapolate deductions when the 
observed defects equaled the CND. 

The instructions associated with line "I" at San 
Francisco leaves no clue as to what the actions should be 
when the observed defects equal the CND. In the one 
instance where the situation occurred, deductions were not 
extrapolated when the observed defects equaled the CND. 
Contract administrators were unaware of the proper procedure 
to follow in this case. Clearer instructions should advo- 
cate extrapolation when the, "observe defects are equal to 
or greater than the CND." 

3. Treatment of Ligqguidated Damages 

Liquidated damages were extrapolated at both 
activities (see Figures 5.4 and 5.5). The SCM at Great 
Lakes justifies this action by claiming that the customer 
suffers damages through the failure of the contractor to 
perform. This issue remains unresolved. In their new 


contracts, San Francisco will continue to extrapolate 
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Figure 5.5 Exterior Service Payment at FWC San Francisco 
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liquidated damages, while Great Lakes will claim liquidated 
damages only for work which is re-inspected. 

A firm policy needs to be established for each 
category of work. Where a government function is the recip- 
ient of the service, as in a janitorial contract, extrapo- 
lated damages may be more easily supported. This is a 
different situation than attempting to assess damages for 
the housing occupants' inconvenience. In any case, the 
extrapolation of liquidated damages can be used as extra 
incentive to obtain contractor compliance with the contract. 

4. Treatment of Validated Customer Complaints 

Both activities took deductions for validated 
customer complaints in addition to extrapolation. This 
procedure 1S incorrect. The extrapolation of deductions 
theoretically accounts for all deficiencies which exist 
outside the sample. The most advantageous way to treat 
validated customer complains under the field version is to 
take deductions unless the CND has been reached. As can be 
seen in Figure 5.5, PWC San Francisco makes no_- such 
distinction. 

5. Deduction Data 

Extrapolation occurred ten times at PWC Great Lakes 
and only once at PWC San Francisco, although each lost an 
additional opportunity to extrapolate when the observed 
defects equaled the CND. The difference can be attributed 


mainly to the lower CPL used at San Francisco. However, 


col 


it's the potential of extrapolated deductions that is being 
counted on to improve contractor performance. Greater 
deductions are not being sought for their own sake. This 
potential is evident by analyzing the nature of the 
deductions. 

A total of $19,195.00 was taken in service call 
deductions over the 24 months evaluated. That represents 
2.1% of the contracted price for these services. Forty-five 
percent of those deductions, $8,694.00, were attributable to 
extrapolation beyond observed defects. 

While only one incident of extrapolation took place 
in eleven months at San Francisco, it resulted in deductions 
of $2,731.00 above the observed deductions. That represent- 
ed 23% of the total deductions of $11,768.00. Total 
deductions at San Francisco amounted to a reduction in the 
contract price wer Ilaac- 

Thus it can be seen that a contractor certainly has 
reason to avoid extrapolation. When only a small portion of 
a contract is being inspected the number of observed defects 
will also be small. Therefore, deductions taken for 
observed defects will be minor. Where the contractor can 
also expect to rework the item, there is little financial 
incentive to do it right the first time. However, if 
deductions are extrapolated, the final percent deducted will 
more closely reflect actual performance. The impact of this 


is shown graphically in the next section. 


78 


6. Effect on Performance 

It is impossible to definitively state what factors 
influenced performance at each activity. One fact is that 
both contractors quickly learned to avoid extrapolation. 
The ten incidences of extrapolation at PWC Great lakes 
occurred in the first eleven months of the contract. Seven 
of those extrapolations were assessed for unsatisfactory 
performance in response to routine calls. Response to 
routine calls was the only area where Great Lakes experi- 
enced any problems. 

Figure 5.6 plots contractor performance in response 
to routine calls at Great Lakes, along with the correspond- 
ing percent of contract payment each month. Extrapolation 
took place from June to October; and then again in December 
and February. It should be noted that in these months 
payment actually fell below the performance level. This is 
an undesirable state of affairs from NAVFAC's point of view, 
and occurred because deductions were taken for validated 
customer complaints in addition to the extrapolation. Due 
to the high CPL of 97%, the contractor was forced to perform 
within a relatively narrow band to avoid extrapolation. In 
February of 1985, for example, the CND was exceeded even 
though performance was at 95.4%. The last year of the con- 
tract has seen steady high performance in all service areas. 

Performance at PWC San Francisco was erratic in all 


service categories. Figure 5.7 plots performance it 
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exterior maintenance, along with the corresponding percent 
of contract payment each month. Except for the extrapola- 
tion which occurred in the first month, performance in 
exterior maintenance typified the other two work categories. 
The potential of extrapolated deductions is most evident 
from this plot. It can be seen that where extrapolation 
took place, payment was well below performance. Again, this 
is a result of taking deductions outside the _ sample. 
However, due to the low CPL of 90%, the contractor was able 
to perform at levels as low as 85.7% without exceeding the 
CND. Because the level of inspection was low, large gaps 
existed between payment and performance. In fact, the 
closer the contractor could get to the CND without reaching 
it, the greater would oe his favorable gap between perform- 
ance and payment. Thus, it appears that the contractor was 
able to manipulate his performance to maximize payment 
differential. 
7. Comparison with RSED V3.0 

I did a deduction analysis with the PWC San 
Francisco data assuming that RSED V3.0 had been in affect. 
The questionable assumption in such an analysis was that 
contractor performance remained the same. This is question- 
able because performance would have been judged unsatsifac- 
tory eight times vice once, resulting in more pressure 


through CDRs. In addition, the fact that payment would have 
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more closely matched performance would have _ provided 
additional incentive. 

The level of deductions under RSED V3.0 would have 
been $3,766.00 per month. This represents 4.7% of the 
contract price and a monthly increase in total deductions 
over the field method of $2,696.00. This increased deduc- 
tion level would have to be paid for with increased inspec- 
tions. Monthly inspections required by RSED V3.0 would 
represent an 81% increase from 221 inspections to 401. At 
least one new inspector would have been required to cover 
this requirement. The extra deductions would pay for that 
new inspector only if performance did not increase! 

Discrepancies between pay and performance will be 
minimal at all levels under RSED V3.0. Payment will 
normally be about 1% in excess of performance if this 
methodology is properly applied. This will eliminate any 
motivation for the contractor to underperform, as now exists 
at San Francisco. However, the fact that all deductions 
will be extrapolated removes a clear goal tied to the 
pocketbook. That "goal" appeared effective in ensuring 
contractor performance within a narrow band at PWC Great 


Lakes. 
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VI. CONCLUSIONS AND RECOMMENDATIONS 


A. GENERAL ISSUES 
1. Need for Oversight 

This study found that ina few areas, discrepancies 
existed between the RSED procedures implemented at the test 
activities and the intentions of NAVFAC. These discrepan- 
cies occurred because of either a lack of clear guidance or 
an omission on the part of the activity. Major examples 
indicating a lack of communication between NAVFAC and the 
activities are: (1) the failure of both activities to 
extrapolate when observed defects equaled the CND, (2) the 
taking of deductions in addition to extrapolation, and (3) 
different approaches planned for the extrapolation of 
liquidated damages. 

Problems such as these must not only be anticipated 
in future guidance, but incorporated into sample working 
documents where possible. In addition, the potential for 
misunderstandings justifies an internal command assistance 
program. Evaluation of RSED plans at the EFD level would 
promote uniformity and provide assistance. This simple 
followup would help preclude the Air Force experience with 
GAO audits. Because of the small number of activities 


involved, effective oversight would not be difficult to 
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implement. When correct procedures became entrenched in the 
system, this special effort could be phased out. 
2. Treatment of Sample Adjustment 

The review of activity QA plans and discussion with 
contract administrators indicated that the potential for 
unnecessary sample adjustment was great. Current guidance 
does not clearly address the issue of sample size adjust- 
ment. This problem becomes even more critical under RSED 
V3.0 due to the monthly extrapolations and narrow sampling 
requirements. Lacking statistical backgrounds, contract 
administrators' actions are unpredictable. Worse than the 
expedient procedure at PWC San Francisco, would be to 
overcompensate. 

The analysis of the effect that sample size had on 
confidence level suggests that original samples’ should 
provide adequate basis for contract action in all but the 
most extreme cases. A simple cutoff might be a minimum 
Sample size of 50. Where the original sample is based on a 
good faith estimate and exceeds 50, no adjustment would be 
necessary. Specifying a minimum confidence level of 50% 
before adjustment was necessary would be another approach. 
However, that would require software support to speed the 
process and ensure against miscalculation. 

3. Combining Homogeneous Populations 
The benefits of random sampling increase exponen- 


tially with population. Due to the nature of service 
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contracting, strict homogeneity is almost never achieved 
within any population. Where two services have the same CPL 
and similar unity costs, the possibility of combining those 
populations should be examined. Inspection to different 
standards would not present a problem. This process would 
be similar to the combination of service calls into another 
category as was done at San Francisco. However, in that 
case no benefits were realized because three categories were 
still used. Candidates for combination should be studied 
and methods promulgated to make RSED more efficient. 
4. Surveillance Levels 

The wisdom of sampling at increased levels of 
inspection is questionable. Tightened inspection does not 
result in increased levels of extrapolation. Sample sizes 
required by tightened inspection (99% confidence) are Signi- 
ficantly higher than normal sampling. It was done at PWC 
Great lakes through the use of overtime and reassignment. 
At PWC San Francisco, constrained resources prevented its 
use. 

Where contractor performance has been poor and 
inspector resources are available, minimum levels of random 
sampling should be supplemented with planned sampling. This 
procedure makes use of inspector judgment to enhance the 
probability of identifying defects while maintaining 
extrapolation. More flexibility is gained because there is 


no need to obtain a "proper" sample at the higher level. 
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Future guidance should clearly document the procedures for 
mixing these two QA techniques and promote its use. 
5. Performance/Payment Match 

Under the field version, the mismatch between 
payment and performance increases with a lower CPL. While 
this mismatch provides a powerful incentive to avoid 
extrapolation, it appeared to result in planned nonconforn- 
ance at PWC San Francisco. Deductions can only act as a 
motivator where payment closely follows performance. 

Guidance specifications, when developed, should 
include minimum CPLs. This is especially true where the 
field version iS in use, aS inspection levels are a function 
of the CPL. Activities should be required to justify lower 
CPLs. This study indicates that a CPL of 90% for service 
calls is too low. 

6. Computer Assistance 

The complexity of RSED requires computer support 
over a range of operations. This support is necessary to 
both reduce the administrative burden and minimize the 
chance of error. Programs now exist which are close to 
satisfying this need. Full service programs should include: 
sample generation with flexible numbering, sample adjustment 
with decision routine, payment analysis, and sample history 
if appropriate for identifying planned samples. 

Computer support should be a requirement for activi- 


ties planning on implementing a RSED program. The high 


86 


payback of random sampling would justify this investment. 
Software issued by NAVFAC could also be designed to help 


ensure uniformity in the application of RSED. 


Be. RSED V3.0 

The proposed method of taking extrapolated deductions is 
an improvement over the field version in that deductions and 
inspection requirements are not a function of the CPL. It 
provides a better match between payment and performance at 
lower CPL levels. The sample sizes required by RSED V3.0 
are approximately equal to normal sampling at a CPL of 95% 
and reduced sampling at 97%. Thus, activities with a 
planned CPL of 90% or less would experience a significant 
increase in sample requirements. 

Possible disadvantages associated with RSED- V3.0 
include: 


(1) RSED places more emphasis on proper samples due to 
monthly extrapolation. 


(2) There is no clear measure of good performance which 
provides financial incentive. 


(3) Where contractor performance is consistently good, 
there is no reduced inspection level possible because 
of the need to extrapolate. 


(4) Influence of customer complaints is diminished 
because they cannot be used to take deductions. 


(5) The addition of a payment "adjustment" is one more 
complication added to the payment analysis. Rounding 
procedures must be clearly addressed. 


The importance of these disadvantages was not studied. 
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I support NAVFAC's intentions to try RSED V3.0 because 
of the weakness of the field version in influencing 
performance at low CPL levels. This phenomenon was demon- 
strated at PWC San Francisco. However, results from the 
field will have to be analyzed before a definitive statement 
can be made. Interesting comparisons could be made by 
shifting PWC San Francisco and PWC Great Lakes to RSED V3.0 


under the same contractor. 


C. RECOMMENDATIONS FOR FUTURE STUDY 


1. The impact of RSED V3.0 on contract price and 
performance. 


2. Relationship between inspector aggressiveness and QA 
technique. For example, if an inspector knows that 
every defect identified will have ramifications beyond 
its own importance, will he be less likely to document 
borderline cases? 


3. <A cost/benefit study of RSED versus planned sampling. 


4. A legal study which attempts to define the parameters 
within which NAVFAC must stay when designing a RSED 


program. 

5. Identification of specific contractors which serve 
both the Navy and Air Force under their respective 
RSED programs. Compare payment and performance. 


Interview the contractor for his perspective. 
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APPENDIX A 


USING MIL-STD-105D 


The procedure contained in this appendix is taken in 
part from OFPP Pamphlet No. 4, "A Guide For Writing and 
Administering Performance Statements of Work For Service 
Ssomceracts." The OFPP approach is equivalent to Air Force 
inspection procedures given in AFR 400-28. A sample size 
and corresponding accept/reject numbers are then determined 
for a hypothetical case, along with the associated 


producer/consumer risk. 
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SAMPLING PLAN 


A. Deciding on the Acceptable Quality Level (AQL). The AQL 
is the highest number of defects per hundred, highest 
percent defective or highest number of defects that can be 
allowed for any service performance indicator. There are 
only a limited number of AQLs listed in MIL-STD-105D but, in 
virtually all cases, one will be close enough to control the 
contractor's level of service. 


(1) The first step in designing a sampling plan under 
MIL-STD-105D is the selection of a realistic AQL. No 
service can be perfectly performed. 


(2) Find the closest AQL from Figure 1 and use it to 
replace the original AQL on the Performance Requirement 
Summary. For example, the AQL for taxi service might have 
been 5 percent. This would be changed to 4 percent or 6.5 
percent since 5 percent does not appear in the figure. 


B. Determining the Lot Size. To determine the sample size, 
the lot size must be known. The lot is how often the 
contractor provides the service ina period of time. 


(1) To determine the lot size, estimate the frequency 
of the service to be sampled, during the period it is to be 
sampled. 


(2) In the case of workorders, the monthly lot size can 
be estimated from historical information on file. 


C. Determining the Sample Size. Use Figure 2 to identify 
an appropriate sample size for a given lot size. 


(1) Use the normal sample size column unless there is a 
limited number of QAEs or unless the cost of an inspection 
Suggests the use of the medium or small sample size column. 


(2) Use the medium or small sample size, if inspec- 
tions for a particular service are lengthy or hinder the 
contractor's ability to provide service to customers. 


D. Selecting the Rejection Level. Use MIL-STD-105D to 
identify the acceptance and rejection level for the sample 
size (see Figure 3). To use the figure, begin with the 


known values for the AQL and the sample size. 


(1) Find the selected sample size (in the sample size 
column) and read across that line to the column for the 
selected AQL. At that point there will either be two 


numbers or an arrow pointing up or down. 


90 





Allowable Acceptable Quality Levels 





0.030 % 1.0 2 
Q.015 % 1.2) = 
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0325 3 PA 
0.40 % 40. % 
0.65 % 65. % 


Figure 1. List of MIL-STD-105D Acceptable Quality Levels 
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Figure 2. Sample Sizes 
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(2) If there is an arrow, follow the direction of the 
arrow until it leads to a pair of numbers. Of the two 
numbers at the intersection or at the end of the arrow, the 
number on the left (Ac or accept) indicates the maximum 
number of defects which can occur in a sample and still 
permit the total group or lot to be judged acceptable. 


(3) When there is no accept or reject number for a 
given sample size and AQL, following the arrow will also 
cause a change in sample size. For example, with an AQL of 
1.5 and a sample size of 20, the sample size would become 
32. 


(4) The number on the right (Re or reject) indicates 
the minimum number of defects that occur in a sample which 
causes the total group or lot to be judged unacceptable. 
For example, suppose the sample size is determined to be 32 
and the AQL has been set at 6.5 defects per hundred. Find 
the number 32 in the sample size column and read across that 
line until the AQL column for 6.5 has been reached. The two 
numbers at that intersection are 5 and 6. 
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EXAMPLE CALCULATION 


Given information: 


1. Estimated number of services for month: 1000 
Zee AQL:?: 6.5 percent 
3. Normal inspection desired 


Determined from the tables: 


1. Sample size: 80 
2. Accept number: 10 
3. Reject number: 11 


It should be noted for this example that even though the 
activity has specified 6.5% defective as satisfactory 
performance, it will accept 10 defective work units out of a 
sample of 80!. This equates to 12.5% of the sample being 
defective. 


From Figure 4 it can be seen that if the actual level of 
defects present were 8.0%, the probability of acceptance 
would be 95%. This equates to a producer (contractor) risk 
of 5%. At the same time, even if the actual level is as 
high as 11%, it will still be accepted 75% of the time. 
Thus consumer (government) risk at that level of performance 
is 75%. This represents a large bias in favor of the 
Seontractor. 
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CHART C-G - OPERATING CHARACTERISTIC CURVES FOR NORMAL-TIGHTENED SINGLE SAMPLING SCHEMES 
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Figure 4. Operating Characteristic for Sample Size=32 
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APPENDIX B 


MO-327 RANDOM SAMPLING PROCEDURES 


This appendix is an excerpt from the current NAVFAC MO- 
B27 entitled "Service Contracts: Specifications and 
Surveillance." It represents the NAVFAC guidance for random 
sampling prior to the field testing of extrapolated deduc- 
e2ons. The MO-327 is presented here to illustrate the 
limited random sampling guidance available to activities 
planning to begin a random sampling program. Along with an 
explanation of random sampling, this appendix includes 
inspection tables and a procedure to generate random 
numbers. 

When NAVFAC has finalized their procedure for implement- 
ing extrapolated deductions Navy-wide, the guidance will 


appear in a new MO-327. 
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RANDOM SAMPLING 


1. GENERAL. Random sampling is a surveillance method based 
on statistical theory. The key element of random sampling 
is that each and every occurrence of work has an equal 
chance of being evaluated. In order to achieve the desired 
end results, knowledge Or the Contractor’s overall 
performance based on evaluation of only part of the work, 
surveillance by random sampling must be applied properly. 
Key elements of random sampling are: 


a. Sample size is specified for a given population to 
achieve a predetermined level of statistical accuracy. 


b. The sample of work occurrences selected for 
evaluation must be selected by a random process in which 
each occurrence has equal chance for selection. 


c. Once an evaluation schedule has been established, it 
must be followed through the surveillance period (i.e., 
monthly schedules). 


ad. Surveillance data gathered by other methods (i.e., 
customer complaints and unscheduled inspections) can not be 
combined with data gathered by random sampling. 


e. Assessment of the Contractor's overall performance, 
projected from the observed condition of the sample, will 


always have the potential to be in error. Statements as to 
overall performance should be stated as "The Contractor's 
overall defect rate is in excess of Kaeo (Where "Xx" 


equals the observed defect rate minus one half the AQL--ex. 
ODR (9.2%) - 1/2 AQL (10%) = 4.2%.) 


2. MECHANICS OF RANDOM SAMPLING. Random sampling is a 
structured approach based on- statistics to contract 
surveillance. As such, there is a set procedure in its 
application. The mechanics of applying random sampling are 
as follows. 


3. POPULATION. The total number of work occurrences for a 
given function that are to be performed during the surveil- 
lance period must be known or accurately estimated. The 
Inventory of Service Worksheet is used to determine popula- 
tion size. 


a. When work is scheduled, population size is easy to 
determine. 


Example: Activity xX has 80 dumpsters, 70 are 
emptied weekly and 10 are emptied twice a week. The 
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population, total number of work occurrences per month, is 
Been (/0 x 4) + (10 x 4 2). 


b. When work is unscheduled, population size must be 
estimated. 


Example: The number of service requests for the 
past six months were: 


Jan 321 
Feb 301 
Mar 295 
Apr 337 
May 340 
JUTE 2.0) 
Total 1914 


The average number of service calls per month has been 319 
(1914/6). This would be the expected population for service 
calls for next month unless there 1s some known reason to 
expect a change. 


4. SAMPLE SIZE. Sample size requirements are based on AQL, 
population size, and level of surveillance. Sample size 
tables are used to determine sample size. Tables for normal 
Surveillance, reduced surveillance, and increased surveil- 
lance are attached. 


a. Select the table with the desired level of surveil- 
lance (Tables I, II, or III). 


b. Select the column with the required AQL (.05, .10, 
meow. 20, Or .25) 


c. Select the row that is closest to the population 
size, preferably the next largest entry. 


ad. The number indicated by the row and column selection 
is the sample size required for surveillance in one surveil- 
lance period. 


(1) Daily surveillance requirements will be deter- 
mined by dividing the required period's sample size by the 
number of days that surveillance is to be conducted. 


(2) Weekly surveillance requirements will be 
determined by dividing the required sample size by the 
number of weeks in a period. 

(3) When computing weekly or daily sample sizes, 
always round up to the next whole number (e.g., 45 monthly 
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samples required and 20 work days per month results in a 
daily sample size of 3 - (45/20) = 2.25 and rounded up 
results to 3). 


5. Sample Selection. The final thing to be decided in 
sampling is how the sample will be drawn. The objective in 
the method is to insure that the sample is random (that is, 
that all services have an equal chance of being selected). 
To achieve random selection, use a random numbers table as 
explained in the following examples. (A random numbers 
table, Table IV, is attached). Most items will fall into 
one of these examples. 


a. Use of The Random Numbers Table. The random numbers 
in Table IV are arranged in groups of two. 


(1) To use the table, begin by picking at random a 
group of numbers on any page of the table. This is usually 
done by closing the eyes and pointing with a pencil or 
finger to some initial group. 


(2) To identify additional random numbers, follow a 
pattern. Go along a given line to its end and then along 
the next line to its end and so on through the table until 
enough numbers have been selected or until the table ends. 


(3) If the table ends and there are still more 
numbers to select, go back to the beginning of the table and 
continue using the same pattern. Use various patterns 
alternately, for example, use lines for one sample, use 
columns for the next sample, and use a diagonal pattern for 
the third sample. 


b. How To Use the Random Numbers Table To Identify a 


Random Sample of Consecutively Numbered Work orders. 
Suppose one has to identify a random sample of 97 work 


orders for evaluation. (Sample size is based on a 
population of 319 using normal surveillance.) This can be 
done at the beginning of the month (before the work orders 
are written) or at the end of the month. 


Gi) If there are, or might be, 319 consecutively 
numbered work orders to select from, then one begins by 
listing the lowest work order number (known or projected). 
This cold be $001, or possibly 443, or any other sequen- 
tially assigned number. List the highest work order number 
(known or projected); in this case, it could be #319 or 762. 
For this example, use work orders numbered #443 to 762. 


(a) Select 97 three digit numbers from Table IV 
using a consistent pattern. 
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(ob) If random numbers selected are not between 
443 and 762, discard the number outside the designated range 
and select a new number. 


(2) For example, using the initial entry on Table 
IV we would select number 441. This number is too low. The 
next number, going down the column, is 343 again too low. 
The third number selected is 749. This number falls in the 
range of work order number (443-762) subject to inspection. 
So work order number 749 is selected to be inspected. The 
next work order number selected is 523. 


(3) This process would be continued until three 
work orders are selected. 


c. How To Use The Random Numbers Table To Identify A 


Random Sample From A Group of Items. If a number of items 
need to be sampled that are not consecutively numbered, the 


Simplest solution is to list the identifiers, for all the 
items in a column, on a piece of lined paper. 


(1) Next, number the lines consecutively, beginning 
with the number one. Now use the random number table to 
draw the sample from the line numbers. A selected line 
number leads to the identifier located on that line, and 
that identifier tells which item to sample. For example, if 
one chooses to sample a set of work orders with attached 
Sales slips, one is not going to have to have a set of 
consecutively numbered work orders because not every work 
order has a sales slip attached. 


(2) List the work orders with sales slips in a 
column, number each line in the column, and randomly select 
enough line numbers to make up the sample. 


a. How To Use The Random Numbers Table To Identify a 
Random Samples of Days. Suppose one wants to identify four 


days in the month on which to sample something. The days of 
the month can be numbered 01 to 31 (or less, as 
appropriate). 


(1) It is best to use a starting point different 
from the one used in the previous example. For the purpose 
of this example, it is being used again. 


(2) One can move down the column from number to 
number until the first number between 01 and 31 is spotted. 
In this case, it is 22. Thus the 22nd day of the month is 
selected for sampling. 


(3) Continuing in this fashion, one discovers that 
11 is the next number selected. This number is disregarded. 
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Proceed in this manner until the four days for sampling have 
been identified. In our example, the 4 days selected would 
be 22, 11, 10 and 24. 


(4) If it is not desirable to sample on weekends, 
discard those days selected that happen to fall on a weekend 
and continue that selection until the proper number of days 
has been selected. 


e. How To use the Random Numbers Table To Identify a 


Random Sample of Times of Day. If one wants to select 
random times of day to sample a service such as taxi or bus 


service, use the 24 hour clock. 


(1) If there are any constraints during each 24- 
hour period, take them into consideration. For example, 
suppose that base bus service operates between 0700 and 
2300. Convert these times to minutes (e.g., 0700 = 0, 0410 
= 130, 12155 =315- Vere. Again, uSing Table IV and 
selecting three digit numbers and proceeding across the line 
from the initial number, one comes to 441, or 1421 hrs, as 
the first random time. 


(2) The next random number is 343, or 1243 hrs. 
The number is good and so one schedules an observation for 
1243 hrs. 


(3) Proceed in this manner until the desired number 
of sample times have been identified. 


f. How To Insure Variety in the Use of the Random 
Numbers Table. The use of variety in the random number 


table ensures that detectable patterns do not occur. 


(1) Success in using the tables requires 
consistency but also variety. The above information should 
ensure that the tables are properly used and that the sample 
1s randomly drawn. 


g. Other Random Numbers Generating Methods. The use of 
a hand held calculator with a random number generating capa- 


bility is an alternative to the use of random numbers table. 
Using this type of calculator the QAE would enter the 
minimum value and maximum value and numbers generated would 
always be within the desired range. 
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400 
425 
450 
475 
S00 
550 
600 
650 
700 
780 
800 
850 
900 
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1300 
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2500 
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3500 
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4500 
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950 
1000 
1100 
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260 
262 
269 
274 
277 
2380 
282 
294 
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a9 
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99 
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Lil 
113 
alé 
115 
117 
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120 
121 
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125 
126 
127 
128 
128 
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AQL 
a 
ae 
40 
46 
Ss] 


POPULATI 
Sizes 
50 
75 
100 
125 
1590 
175 
200 
225 
250 
275 
300 
325 
350 
375 
400 


450 
475 
$00 
5590 
600 
650 
700 
750 
800 
850 
900 
950 
1000 
1100 
1200 
1300 
1400 
15090 
1600 
1700 
1800 
1900 
2000 
2500 
3000 
3500 
4000 
4500 
$000 
6000 
7000 
8000 
9000 
+9000 


SAMPLE SIZE REQUIREMENTS FOR INCREASED SURVESLLANCE 
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324 
334 
344 
332 
361 
368 
376 
389 
401 
411 
421 
429 
437 
444 
451 
457 
462 
485 
§01 
213 
523 
$30 
a3 7 
$46 
354 
$59 
$63 
$67 


220 
43 
$9 
74 
87 
98 

108 
116 
124 
132 
138 
144 
150 
155 
259 
164 
168 
Myles 
175 
178 
184 
189 
194 
198 
202 
206 
209 
212 
214 
217 
221 
225 
228 
231 
234 
236 
238 
240 
241 
243 
249 
253 
256 
259 
260 
262 
263 
266 
267 
268 
269 
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144 
146 


165 


1ll 
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SHORT TABLE OF RANDOM NOMBERS 


63 
24 
65 
86 
03 
$6 
93 
4a 
34 
47 
09 
80 
38 
$4 
81 
72 


87 
81 
a 
oo 
24 
47 
82 
74 
a 
ye 
$4 
34 
LO 
28 
61 
03 
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67 
30 
7a 
60 
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APPENDIX C 


SAMPLING TABLES USED 


IN THE FIELD TEST 


This appendix contains tables reproduced from those 


utilized at each activity. The first group of tables were 


used by PWC San Francisco at 
Lakes used the second group 
level. Both groups of tables 


formula. The tables provide 


the CPL = 90% level. Great 
of tables at the CPL = 97% 
were generated from the same 


graphic illustration of the 


effect that the CPL, surveillance level and population have 


on the sample size and CND. 
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CND/SS TABLE 


CRITICAL NUMBER OF DEFECTIVES (CHD)/SAMPLE SIZE (SS) 


Reduced Surveillance (Page‘l of 2 
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= > a 
> UNITS > GREEICALD PERFORMANCE CEVEL (CPL) 

> OF WORK > > > > > > 
> POPULATION > 95% > 90% > 85% > 80% > ie > 
> a2 oe > > > > > > 
> > CHD/SS > CYD/SS > CID/SS > C3yD/SS > Gau/ss > 
> 50a > 3/36 > S727 > 6/21 > 6/17 > 6/14 

> 75 > 4/4? > 6/33 > 6/25 > 7/19 > 7/26 

> 100 > $/36 > 6/37 > 7/27 > Usa > 7/16 

So 125 > 6/62 > 7/280 > 7/29 > he > THAT 

> 150 > 6/68 > 7/42 > 8/30 > Lie > 7/17 
Seely 5 > 6/73 > 3/44 > 8/31 > 8/23 > 8/18 

> 200. > 7/77 > 6/46 > 8/31 > 8/23 > 8/18 

> 228 > 7/80 > 8/47 > 8/32 > 8/24 > 8/18 

> 250 > 7/83 > 8/48 > 8/32 _> 8/24. > 8/18 

> 275 > 7/86 > 8/49 > 8/33 > 8/24 > 8/18 

> 300 > 8/88 > 8/49 > 8/33 > 8/24 > 8/19 
See. 5 > 8/90 > 9/50 > 8/33 > 8/24 > 8/19 

> 350 > 8/92 > 9/51 > 9/34 > 8/24 > 8/19 

me 75 > 8/94 > 9/51 > 9/34 > 67275 > 8/19 

> 400 > 8/95 > 9/52 > 9/34 > 8/25 > 8/19 

> 42§ > 3/97 > 9/52 > 9/34 > 8/25 > 8/19 

> 450 > 8/93 > 9/52 > 9/34 > 8/25 > 8/19 

> 475 > 9/99 > 9/53 > 9/35 > 8/25 > 8/19 

> $500 > 9/100 > 9/53 > 9/35 > 8/25 > 8/19 

>  §50 = 9/102 > 9/§3 > 9/35 > 8/25 > 8/19 

> 600 > 9/103 > 9/54 > 9/35 >. 8/25 > 8/19 

> 650 > 9/105 > 9/54 > 9/35 > e725 > 8/19 

> 700 > 9/106 > 9/55 > 9/35 > 8/25 > 8/19 

> 750 > 9/107 > 9/55 > 9/35 > 8/25 > 8/19 

> 800 > . 9/108 > 9/55 > 9/36 > 8/25 > 8/19 

> 850 > 9/109 > «9/85 > 9/36 > 9/26 > 8/19 

> 900 > 9/1210 > 10/56 > 9/36 > 9/26 > 8/19 Z 
> 950 > 9/110 > 10/56 > 9/36 > 9/26 > 8/19 > 
> 1000 > 10/112 > 10/56 > 9/36 > 9/26 > 8/19 > 
> 1100 > 10/112 > 10/56 > 9/36 > 9/26 > 8/19 > 
> 1200 > 10/113 > 10/56 > 9/36 > 9/26 > 8/19 > 
> 1300 > 10/114 > 10/57 > 9/36 > 9/26 > 8/19 > 
> 1400 > 10/115 > 10/57 > 9/36 > 9/26 > 8/19 > 
> 1500 > 10/115 > 10/57 > 9/36: > 9/26 > 8/19 > 
> 1600 > 10/116 > 10/57 > 9/36 > 9/26 > 8/19 > 
> 1700 > 10/116 > 10/57 > 9/36 > 9/26 > 8/19 > 
> 1800 > 10/117 > 10/57 > 9/36 > 9/26 > 8/20 > 
> 1900 > 164117 > 10/57 > 9/37 > 9/26 > 3/20 > 
> 2000 > 10/118 > 10/57 > 9/37 > 9/26 > 8/20 > 
> 2500 > 10/119 > 10/58 > 9/37 > 9/26 > 8/20 > 


CHD/SS TABLE 


CRITICAL NUMBER OF DEFECTIVES (CHD)/SAMPLE SIZE (SS) 
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> > > 
> UMITS > ¢RITICAL PERFORMANCE LEVEL (CPL) 
> OF WORK > > > > > > 
> POPULATION > 95% > 90% > 85% > 80% > 75% > 
> SIZE > > > > > > 
> > CHD/SS > CHD/SS > CHD/SS > CHD/SS > CND/SS > 
> 3000 > 10/120 > 10/58 > 9/37 > 9/26 > 8/20 > 
> 3500 > 107121 > 10/58 > 9/37 > 9/26 > 8/20 > 
> 4000 > 10/121 > 10/58 > 9/37 > 9/26 > 8/20 > 
> 4500 > Weaver > 10/58 > 9/37 > 9/26 > 8/20 > 
> $5000 > 10/122 > 10/58 > 9/37 > 9/26 > 8/20 > 
> 6000 > LOvUZ2 > 10/59 > 9/37 > 9/26 > 8/20 > 
> 7000 > 1/725 > 10/59 > 9/37 > 9/26 > 8/20 > 
> 8000 > Oa ya > 10/59 > 9/37 > 9/26 > 8/20 > 
> 9000 > Ti/iZ3 > 10/59 > 9/37 > 9/26 > 8/20 > 
> 10000 > div ieo > 10/59 > 9/37 > 9/26 > 8/20 > 
> 20000 > 11/124 > 10/59 > 9/37 > 9/26 > 8/20 > 
> 30000 > 11/124 > 10/59 > 9/37 > 9/26 > 8/20 > 
> 40000 > Eis izo > 10/589 > 9/37 > 9/26 > 8/20 > 
> _§0000 > 11/125 > 10/59 > 9/37 > 9/26 > 8/20 > 
> 60000 > 7 25 > 10/59 > 9/37 > 9/26 > 8/ 20 > 
* 70000 > Vela > 19/59 > 9/37 > 9/26 > 8/20 > 
> 80000 > LiviZs > 10/59 > 9/37 > 9/26 > 8/20 > 
> 90000 > Wsi2o > 10/59 > 9/37 > 9/26 > 8/20 > 
> 100000 > 227225 > 10/59 > 9/37 > 9/26 > 8/20 > 
>. 250000 > W2A125 > 10/59 > 9/37 > 9/26 > 8/20 > 
> §00000 > cai 25 > 10/59 > 9/37 > 9/26 > 8/20 > * 
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > >* 
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > 
> °> > > > -> > 
> > > > > > > 
> > > > > > > 
> > > > > > > 


we, 


AO 


CND/SS TABLE 


CRITICAL WUMEER OF DEFECTIVES (CND)/SAMPLE SIZE (SS) 


Normal Surveillance (Page 1 of 2 ) 
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> > 
> UNITS > CRITICAL PERFORMANCE LEVEL (CPL) > 
> OF WORK > > > > > > 
> POPULATION > 95% > 90% > 85% > 80% > 75% > 
> SIZE > > > > > > 
> > CHD/SS > CND/SS > CHD/SS > CHD/SS > CyD/SS > 
> 50 > 4/40 > 6/33 > 7/28 > 8/23 > 8/20 > 
>. 75 > $755 > 7/42 > 9/34 > 9/27 > 9/23 > 
> 100 > 6/67 > 8/49 > 10/38 > 10/30 > 10/25 > 
= 125 > 7/78 > 9/55 > 10/41 > 11/32 > 11/26 > 
> 150 > 8/87 > 10/59 > 11/44 > 11/34 > Ti7Zs > 
a LS > 8/95 > 21/63. > 11/45 > 12/35 > Li727 > 
> 200. > 9/101 > 11/66 > 12/47 - > 12/36 > 11/28 > 
> 225 > 9/107 > 11/68 > 12/48 > 12/36 > 11/25 > 
> 250 > 10/113 >. 12/70 > 12/49 > 12/37 > 12/49 > 
oe) 5 > 10/118 > l2772 > 12/50 “> 12/37 > 12/29 > 
> 300 > 10/122 > 12/74 > M75] > 13/38 > 12/29 > 
3.325 > 12/126 > 13/75 > 13/52 > 13/38 > 12/30 > 
> 350 > 11/130 > 13/76 >-- 13/52 > 13/39 > 12/30 > 
>- 375 > i7i133 > WS Are > 13/53 > 13/39 > 12/30 > 
> 400 > 12/136 > 13/78 > 13/§3 > 13/39 > 12/30 > 
> 425 > 32/139 > 13/79 > 13/54 > 13/39 > 12/30 > 
> 450 > 12/141 > 13/80 > 13/54 > 13/39 > 12/30 > 
> 475 > 12/144 > 14/81 > 13/54 > 13/40 > 12/30 > 
> §00 > 12/146 > 14/82 > 14/55 > 13/40 > 12/30 > 
>  §50 > 13/150 > 14/83 > 14/55 > 13/40 > 13/37 > 
> 600 > 13/253 > |. 14/84 > 14/56 > 13/40 > 13/32 > 
> 650 > 13/156 > 14/851 > 14/56 > 13/41 > pA > 
> 700 > 13/159 > 14/86 > 14/56 > 13/41 > 13/31 > 
> 750 > 14/162 > 14/85 > 14/57 > 13/41 > 13/32 > 
> 800 > 14/164 > 15/87. > 14/57 > 13/421 > 13/31 > 
> 850 > 14/166 > 15/87 > 14/57 > 13/41 > 13/31 > 
> 900 > 14/167 > 15/88 > 14/57 > 13/41 > L3/a. > 
> 950 > 14/169 > 15/88 > 14/58 > 13/41 > 13/31 > 
> #1000 > 14/172 >" 15/89 > 14/58 > 13/41 > 13/31 > 
>__ 2100 > 15/173 > 15/89 > 14/58 > 14/42 > asa > 
> 1200 > 15/176 > 15/90 > 14/58 > 14/42 > 13/32 > 
> 2300 > 15/178 > 1§/91 > 15/59 “> 14/42 > 33/32 > 
> 1400 > 15/179 > 15/91 > 15/59 > 14/42 > 13/32 > 
> 1500 > 15/181 > 15/92 > 15/59 > 14/42 > 13/32 > 
> 1600 > 15/182 > 15/92 > 15/59 > 14/42 > 13/32 > 
> 1700 > 15/183 > 15/92 > 15/59 > 14/42 > 13/32 > 
> 1800 > 16/185 > 15/92 > 15/59 > 14/42 > 13/32 > 
> 1900 > 16/186 > 15/93 > 15/59 > 14/42 > 13/32 > 
> 2000 > 16/186 > 15/93 > 15/60 > 14/42 > 13/32 > 
> 2500 > 15/190 > 16/94 > 15/60 > 14/43 > 13/32 > 


CMD/SS TABLE 


CRITICAL NUMBER OF DEFECTIVES (CND)/SAMPLE SIZE (SS) 
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> > > 
> UNITS > CRITICAL PERFORMANCE LEVEL (CPL) > 
> OF WORK > > > > > > 
> POPULATION > 95% > 90% > 85% > 80% > 75% > 
> SIZE > > > > > > 
> > CND/SS > CND/SS > CND/SS > CyD/SS > CND/SS > 
> 3000 > 16/192 > 16/94 > 15/60 > 14/43 > 13/32 > 
> 3500 > 16/194 > 16/95 > 15/60 > 14/43 > 13/32 > 
> 4000 > 16/195 > 16/95 > 15/60 > 14/43 > 13732 > 
> 4500 > 17/197 > 16/95 > 15/61 > 14/43 > 13/32 > 
> §000 > 17/198 > 16/96 > 15/61 > 14/43 > 13732 > 
> 6000 > 17/199 > 16/96 > 15/61 > 14/43 > 13/732 > 
> 7000 > 17/200 > 16/96 > 15/61 > 14/43 > 13/32 > 
> 8000 > 17/201 > 16/96 > 15/61 > 14/43 > 13732 > 
> 9000 > 17/201 > 16/96 > 15/61 . > 14/43 > 13/32 > 
> 10000 > 17/202 > 16/96 > 15/61 > 14/43 > 13/32 > 
> 20000 > 17/204 > 16/97 > 15/61 > 14/43 > 13/32 > 
> 30000 > 17/204 > 16/97 > 15/61 > 14/43 > 13/32 > 
> 40000 > 17/205 > 16/97 > 15/61 > 14/43 > 13/32 > 
> 50000 > 17/205 > 16/97 > 15/61 > 14/43 > 13/32 > 
> 60000 > 17/205 > 16/97 > 15/61 > 14/43 > 13/32 > 
> 70000 > 17/205 > 16/97 > 15/61 > 14/43 > 13732 > 
> 80000 > 17/205 > 16/97 > 15/61 > 14/43 > 13/32 > 
> 90000 > 17/205 > 16/97 > 15/61 > 14/43 > 13/32 > 
> 100000 > 17/205 > 16/97 > 15/61 > 14/43 > 13/32 > 
> 250000 > 17/205 > 16/97 > 15/61 > 14/43 > 13/32 > 
> §00000 > 17/206 > 16.97 > 15/61 > 14/43 > 13/32 > 
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > “> 
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > 

> > > > > > 

> > > > > > 
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CND/SS TABLE 


CRITICAL NUMBER OF DEFECTIVES (CND)/SAMPLE SIZE (SS) 


ightened Surveillance (Page 1 of 2 





> > > 
> UNITS > CRITICAL PERFORMANCE LEVEL (CPL) 
> OF WORK > > > > > > 
> POPULATION > 95% > 90% > 85% > 80% > 75% > 
> SIZE > > > > > > 
> > CND/SS > CND/SS > CND/SS > CHD/SS > CHD/SS > 
> 50 > 4/45 > 7/40 > 9/36 > 1iv3Z > 11/28 > 
>. 75 > 6/63 > 9/54 > 12/47 > 13/40 > 14/35 > 
> 100 > 7/80 > 11/66 > 24/55 > 15/46 > 16/39 > 
> 125 > 8/95 > 13/76 > 15/62 > 17/51 > 17/43 > 
> {250 > 9/110 > 14/85 ° > 17/68 > 187/55 > 18/43 > 
> 175 > 10/123 > 15/92 > 18/72 > 19/56 > 19/47 > 
> 200 > i? iso > 16/99 > 19/76 > 20/60 > 20/49 > 
> 225 > 12/145 > 17/104 > 19/79 > 20/63 > 20/50 > 
> 250 > 13/156 > 18/110 > 20/82 > 21/64 > 21/52 > 
> 275 > 14/165 > 19/114 > 21/85 > 21/65 > 21/53 > 
> 300 > 158/174 > 20/118 > 21/87 > 22/67 > 21/53 > 
> 325 > 15/182 > 20/122 > 22/89 > 22/68 > 22/54 > 
> 350 > 16/189 > 21/125 > 22/91 > 22/69 > 22/55 > 

S79 > 16/196 > 21/128 > 23/92 > 23/70 > 22/55 > 

400 > 17/203 > PPC & > 23/94 > 23/70 > 22/56 > 
> 425 > 18/209 > 22/134 > 23/95 > Rosle > 22/56 > 
> 450 > 18/215 > 22/136 > 24/96 > 24/73 > 23/57 > 
> 475 > 18/220 > 237433 > 24/98 > 24/73 > 23/57 > 
> 500 > 19/226 > 23/140 > 24/99 > 24/74 > 23/58 > 
> §50 > 20/235 > 24/144 > 24/100 > 24/75 > 23/58 > 
> 600 > 20/244 > 24/147 > 25/102 > 25/76 > 24/59 > 
> 650 > Zils 2o2 > 25/150 > 25/103 > 25/76 > 24/59 > 
> 700 > 22/259 > 25/152 > 25/104 > 25/77 > 24/59 > 
> 750 > 22/266 > 257155 > 26/105 > 25/78 > 24/60 > 
> 800 > 23/272 > 26/157 > 26/106 > 25/78 > 24/60 > 
> 8590 > 23/277 > 26/189 > 26/107 > 25/79 > 24/60 > 
> 900 > 23/282 > 26/159 > 26/107 > 25/79 > 24/60 > 
> 950 > 24/287 > 27/162 > 27/109 > 26/79 > 24/61 > 
> 1000 > 24/292 > 27/163 > 27/109 > 26/80 > 24/61 > 
> 1100 > 25/299 > 27/166 > 27/110 > 26/80 > 24/61 > 
> 1200 > 25/306 > 28/168 > 27/111 > 26/81 > 25/62 > 
> 1300 > 26/313 > 28/170 > 27/112 > 26/81 > 25/62 > 
> 1400 > 26/318 > 28/171 > 28/113 > 26/82 > 25/62 > 
> 1500 > 27/7325 > 28/173 > ra Wk 26/82 > 25/62 > 
> 1600 > 27/327 > 29/174 > 28/114 > 26/82 > 25/62 > 
>__1700 > 27/331 > 29/175 > 28/114 > 26/82 > 25/63 > 
> 1800 > 28/335 > 29/176 > 28/115 > 27/83 > 25/63 > 
>__1900_ > 28/335 > 29/177 > 28/115 > 27/83 > "25/63 > 

2000 > 28/341 > 29/178 > 28/116 > 27/83 > 25/63 > 
2590 > 29/353 > 30/181 > 29/117 > 27/84 > 25/63 > 


CND/SS TABLE 


CRITICAL NUMBER OF DEFECTIVES (CND)/SAMPLE SIZE (SS) 
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> > > 
» UNITS > CRITICAL PERFORMANCE LEVEL (CPL) > 
> OF WORK > > > > > > 
> POPULATION > 95% > 90% > 85% > 80% > 75% > 
> Size > > > > > > 
> > CND/SS > CND/SS > CND/SS > CND/SS > CYD/SS > 
> 3000 > 30/362 > 30/183 > 29/118 > 27/84 > 25/64 > 
> 3500 > 30/368 > 30/185 > 29/119 > 27/85 > 25/64 > 
> 4000 > 31/373 > 30/186 > 29/119 > 27/85 > 26/64 > 
> 4500 > 31/377 > 31/187 > 29/119 > 27/85 > 26/64 > 
> §000 > 31/380 > 31/188 > 29/1206 > 27/85 > 26/64 > 
> 6000 > 31/385 > 31/189 > 29/120 > 27/85 > 26/64 > 
> 7000 > 31/389 > 31/190 > 29/121 > 28/86 > 26/64 > 
> 8000 > 31/391 > 31/190 > 29/121 "> 28/86 > 26/64 > 
> 9000 > 31/394 > 31/191 > 30/122 > 28/86 > 26/65 > 
> 10000 > 31/395 > 31/191 > 30/121 > 28/86 > 26/65 > 
> 20000 > 31/403 > 31/193 > 30/122 > 28/86 > 26/65 > 
> 30000 > 31/406 > 31/194 > 30/122 > 28/86 > 26/65 > 
> 40000 > 31/407 > 31/194 > 30/122 > 28/86 > 26/65 > 
> _§0000 > 31/408 > 31/194 > 30/122 > 28/86 > 26/6§ > 
> 60000 > 31/409 > 31/194 > 30/122 > 28/87 > 26/65 > 
> 70000 > 31/409 > 31/194 > 30/123 > 28/87 > 26/6§ > 
> 80000 > 31/409 > 31/194 > 30/123 > 28/87 > 26/65 > 
> 90000 > 31/410 > 31/195 > 30/123 > 28/87 > 26/65 > 
> 190000 > 31/410 > 31/195 > 30/123 > 28/87 > 26/65 > 
> 250000 > 31/411 > 31/195 > 30/123 > 28/87 > 26/65 > 
> §00000 > 31/411 . > 31/195 > 30/123 > 28/87 > 26/65 > 
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > “> 
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > >" > > > > 
> > > > > > > 
> > > > > > > 
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CND/SS TABLE 


REDUCED SURVEILLANCE 


eS 


N 99% 98% G74 96% 9S4 

30 1s 4 27 43 2740 3/38 3/ 3 

oe) 2/ 67 2/6 3793 as asa 

100 aé 6? 3 / V% 4 / of 47 61 3/4 Se 

129 27 109 37 90 orn 9770 Ms 4s 

130 aé iza 4 / iG2 3 / @ & / 77 @ / 68 

173 37 138 @7 4113 3 / % o/ «2. os73 
200 37 193 9.7 123 6 7 103 oso 7477 
223 3 167 9/7 132 6 / 108 7/93 77 
230 3 7 160) 9 7/7 141 @ / 113 ? 4 GF Tse 
273 esi 37 148 6 / 120 77% 100 7s & 
300 47 203 #73139 7s 124 77103 eles 
23 asa? @/ 142 2/129 7 7 106 es 9% 
330 a ze o/ 168 7/132 @/ 109 esa 
373 4/28 6/173 2 / 136 e443 es 9% 
400 9 / 240 0/170 77.199 @7 113 ess 
423 9.7 237 o/163 77142 @vuss evr 
430 3 / 206 7/100 @/ 108 e741? ere 
73 3/279 77192 es 147 8/118 ./ 
300 9 / 783 7s 1% as ue @ / 120 % 7 100 
330 3 20 7 4 203 87 193 9/123 ° 7 102 
&00 o/ 312 77 210 @/ 137 97129. © 7 103 
630 o/s ms gs 33 @/ 160 ° 7 127 * 7 103 
700 o/ 37 @ 7 221 @/ 163 e129 @ / 106 
7390 os 340 e225 °/ lee ° 7 130 ° / 107 
e00 @ / 339 @/ 2306 9 4 1068 9/132 ? 4 106 
530 7/3609 es zx °7/ 170 e133 @ 7 109 

06 7/ 378 es 27 e178 @ 7 136 ° 7 110 
730 7 / mms @s 201 7 174 °7/ 139 #7 110 

1000 74 396 @s 2064 e179 ° / 136 107 488 
1100 7s 409 er 24 7178 es 138 10 4 312 
1200 7 ¢ 422 os 234 e716: 4107129 10 7 133 
1300 @ / 436 °/ 338 7 163 107 141 10 / 1414 
1400 @ s/s 448 ° / 262 10 / 185 10 / 142 10.7 819 
1300 @s 430 ° / 363 10 7 18 10 / 143 107.439 
16400 @/ 43 ° / 208 10 / 180 10 / 144 10 / 186 
1700 es 47 e371 10 / 180 10s 144 10 / 3116 
1800 es 478 ° 7/373 10 / 190 10 7 149 107 417 
1900 es ses es 379 10 7 191 10 / 146 10 / 417 
qos e749) 10.4 377 10 7 192 10 / 146 10 / 138 
2300 e316 10 / 289 10 7 1% 10 / 148 10 7 410 
3000. 9 335 10 7 291 10 / 198 10 7 130 10 / 120 
1906 es 909 10 / 799 10 / 300 10.7 191 10 7 121 
4000 10 7 3960 10 4 20 10 / 202 10 / 132 10 7 1218 
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473 
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gi42223 
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1309 
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CND/SS TABLE 


NORMAL SURVEILLANCE 
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a1 
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ig 
i2 
12 
13 
i3 
14 
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987% 97% 9&% 93% 
2/ «& 2/ 4 3/7 42 asa: 
3/ e& 4/ea ase 37/933 
3 / 64 4 78 3 / T2 & / a7 
as 101 3/ 92 o/ 6 7/70 
asi @ 7 109 7799 es 67 
3 / 132 oy ii7 77 103 es 9 
3 7 148 ? 4 127 @/ 113 9 / 101 
o/ 138 74 337 @/ 121 ° 7 107 
&/ 170 @/ tte 9s 3im 10 / 113 
@/ 101 @/ 1368 °/ 136 10 / 118 
77192 @/ 162 10 7 199 10 / 122 
? / 202 °/ 169 10 / 144 11 / 126 
Peas 9/179 104 149 11 / 1230 
8 / m0 9 / 183 10 / 193 117133 
e/ 22s 10 / 167 11 / 499 12 / 1% 
e/ 2 10 / 192 11 / 161 12/199 
@/ 243 10 / 197 11 / 169 12 / 143 
7 291 10 / 202 11 / 168 12 / 146 
° 7 237 IL / 200 12/171 12 / 146 
9 / 270 1b 7 216 12 / 176 19/190 
10 / 782 3 / 221 12/7 101 13.7139 
10 / 2v2 12 / 22? 13 / 1% 13 / 136 
10 7 302 127 «J3 13 / 109 19/7 199 
117 311 2s 13/193 14 / te! 
117 319 12.7 243 13 / 1% 14 / 166 
13 / 22z7 13 / 248 13/199 14 / lee 
11 / 236 13/4 232 14 / 202 14 / 167 
12 7 340 13 / 236 14 / 206 14 / 169 
12 / 347 13 / 294 14 / 206 147 173 
12 / 338 14 / Jee 14 7 210 19 / 173 
13 / 368 147 271 14 / 216 19 / 17% 
13.7 377 14 / 26 137 217 139/178 
13.7 383 147 280 19/219 i397 179 
13/7 392 14 / 200 19 7 27 19 / 161 
14/7 790 197 287 19 / o23 19 7 142 
147 406 13 / 290 13 / 229 139 / 189 
147 410 137 293 13 / 227 le / 103 
147 418 13 / 2% 19 / a le / 186 
147 419 197 200 13 / z30 le / 186 
19 7 «38 te 7 30? le / 239 le / 190 
13 / 431 l@ 4 333 le / 239 1@ / 1%2 
1@ / 461 1@ 4 318 1@ / 242 1@ / 194 
le / see 16 / 332 le / 246 le / 1% 
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CND/SS TABLE 


TIGHT SURVEILLANCE 
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7e? 


gts 
“ 


916 


7 


gif 


1334 
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98% 7% 96% 937% 
2/4 37 47 37 % as4s 
37 70 as 9 / & o/43 
osu 5 / ee o/ 0 770 
as 43i2 & / 106 7 4 108 as % 
37333 77 334 @s te * 7 330 
3 / 130 7 7 140 7 333 10 / 123 
o/ 108 @/ 136 10 / 144 43 7 333 
7 / 106 ° 7 170 13.7 337 12/169 
? 4 202 10 / 106 12 / 169 13 / 336 
8/218 10 / 197 12 / 180 14 / 363 
es =m 83 / B10 13/19 28/7 374 
© / Dee 33/22 84 / @00 16 / 142 
° / Be wees 16 / 909 Ao / 309 
10 / 277 12 / 246 19/7 218 16 / 3% 
10 / 291 83 7 2399 13 / 22 8? ¢ 203 
10 /¢ 303 13 / 2e6 le / 23 18 / 209 
13 / 336 14 / z78 lo / 243 18 / 233 
13 / 328 16/7 783 17 / 268 18 / 220 
12 7 340 19 / 292 17 / 293 1% / 22s 
12 / 32 le / 208 18 / 27 20 / 235 
13 / 383 16 / 323 19 4 «2779 20 / 248 
147 403 17 / zr 19 / 209 23 / 232 
147 4622 18 / 390 20 / 206 z2 / 230 
1397 479 18 / 342 23 / 207 22 / 266 
13 / 436 1% / 373 21/315 z3/ 272 
1@ / 472 1% / 3me a2 / 223 a3 / 277 
le / 407 20 / 396 2 / x0 2 / 2 
17 7 303 20 / 403 22s De 20 / om” 
17-7933 217 432 23s 342 24 / 292 
1@ / 340 21 / «70 24 / 333 23 / 20 
19 / 302 zz 442 24 / 363 23 / 2% 
20 / 364 aa/ 499 23 / 378 ae / 313 
- 207 s06 23 / 467 23 / 370 26 / 318 
23/7 emd “24/477 26 / 3M 27/323 
21/48 24 / 487 26 / 392 27 / 327 
22 / 633 23 / 4% 26 / 30 27s: 
22 / eos 29 / 306 27 / 403 28 / 3s 
zs 683 2 / 312 27 / 408 as / 3s 
z3/ 63 26 / 91° 27 / 433 aes 341 
23 / 763 27/7 3947 29 / 4230 29 7 393 
26 / 7e08 ze / 308 29 / 443 30 / 3602 
27 / 614 29 / 386 30 / 433 30 / 368 
23 / er 30 / 9% 30 / 460 33 / 373 
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APPENDIX D 


RSED V3.0 TABLES AND PAYMENT ANALYSIS 


This appendix contains inspection tables and deduction 
adjustments (deltas) for RSED V3.0. That is followed by a 
sample payment analysis. The material within this appendix 
has been reproduced from preliminary NAVFAC guidance on the 
proposed method [Ref. 19]. Interesting features are the 
enhanced inspection requirements and narrowly defined 
population ranges. There is no CND because extrapolation 


takes place in the event of any defect. 
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RSED V3.0 - Table of Sample Sizes 


For Normal 


Sampling Levels 


07 -O7-19756 


Sample sizes are for the indicated monthly population: 


Population range —- Sample Size 


SS 
Sy 
S7 
42 
45 
48 
=! 
34 
a7 
el 
54 
&7 
71 
74 
78 
SZ 
B85 
89 
93 
ao 
Holz 
L106 
Lei 
tS 
120 
Zo 
130 
Lo= 
141 
146 
hoZ 
LoS 
164 
170 
ley 7 
les 
190 
Mon 7 
205 
Ze 
220 
Le 
oT 
246 
Z56 
Zoo 
Z73 


30 SS 
32 Se 
34 4li 
56 44 
So 47 
40 30 
42 = 
44 36 
4S a7, 
48 62 
50 &6 
aie 69 
54: TZ 
26 76 
ao BO 
60 84 
62 87 
64 oN 
66 96 
68 100 
70 104 
TE 108 
74 Wis 
76 118 
78 1235 
BO 128 
B2 LSS 
84 138 
BS 143 
88 149 
70 LoS 
TZ eo! 
94 167 
9& 7S 
98 180 
100 157 
Voz 194 
104 201 
106 Oe 
108 216 
110 Zee 
lobes BOs 
114 242 
116 Zot 
ae 260 
120 270 
22 Zot 
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NOS. fcuk 
DOG. sem is 

Pati 1 scenes Gs, 
Bo... ieee 
Bie. <item ; 
a 
PAS. saree 

7 a 
ZOOS. Gees 3 6 
By On Peet oo: 
POS 5. Hage ss 


Population range - Sample Size 


RSED V3.0 - Table of Sample Sizes 
For Normal Sampling Levels 
07-07-1986 


Sample sizes are for the indicated monthly population: 


Population range - Sample Size Population range - Sample Size 
286 - ZT Omer ae Serene 124 291 - 296% s+ so aes 12 
297 - SO? sists ere sore 12 303 - 508.4663 «eee 127 
309 - 5) li Semeaenes care Shee 314 = S208 6 so 6 ee 129 
Be Mle = SOM che ete 130 327 - S26 sere ee 
BEE SS Treen as Ls 340 - 3454s o + «aeeSO 
344 - i) Loreen et cte-o is 134 S25 = So? « «eee yess 
340 = BT ee ee 136 367 - > 1 Bie eet se « eee 7 
S73 = OB le eeeete css ote Va LoS 382 ~ SB. ss tereeme «+ eee 
390 = OT 7st one te ‘emete 140 398 - AOSD. «0 ss « ocean 
4045 - 4A earerexe ers es oa 415 - G22 6 see 0 io cece eee 
4255 — 43 Lice tes + sce 1 ae 432- 440... 6. snueeee 145 
441 - BDO lc eae knee ee 146 4351 - BOT. 6 0 ee 6 6 seer 
460 - BGT aes 2s 5 eet ace 470 - 479 «© sfene eae » 149 
480 - BS Orne cas oles Cees 150 491 - SO1 . 0s « oo seen. 
SIC = SD 12 oe eine @0. cee JS = DL Secee + « oS 
524 - Da) eee oar ote tates 154 356 - 7496. .26 0 0 5 chee 
549 = DOO essere eet 156 261 = D748. ccc we © oe 
5375 - DE 7 shametene ses oc aloe 286 = G01... 0 0 ©  cnenenlee 
502-7 — G1 Gis ctoere ss 2s oe BOO 617 = 631..<0.65 see 161 
652 - 647... iaetsve. 2066 162 648 - G63. .00 6s « eee 
664 - GOO% Brecon erie 164 efsi} = O97 seherene <0 ee eles: 
698 - Pat re rea 166 717 = 15964 see 167 
736 - 7.4 wees aac eee 168 '] a) eee 1126 ete ee 169 
776 - TT Oclcttusne ca oe 2. 1 oO 797 - B19. «cece 6 oleae 
820 - BF 2 ereeen sre tee ss 72 843 ~- S67 wee see 17s 
868 - BS 3 cererereveuss oe 174 894 - O20 «ci enenene eae oo lee 
921 = OG Gore sie ae 176 949 - F7B . «2 cle «ss cues 
979 - 1009 2.2. wee, LS 1010 = 1042.2 cc0cs 6 sce 
1043 - LO7 7 semester et enca 180 [O78 — ELS ieretace oc « eee 

ito VS Sc See eeoreress 182 1154 - L194. . ccc ces eee 
1195 - 123.8 s seeeretenets sens 184 257 = 1285 «. serere +) «ee 185 
1286 - L SSD scmenetseeee sc el oe sé a5 1S86 2.9.0 -o. eee «- UE7 
1389 - 1449 6 sepeecuekers eee 1446 ~- 1507 « « s 6 6 © «een 
1508 = S73. xmemeeere See) eal Oo 1574 - 1644... c00 sss eee 
1645 - 17 2 lVsvememenshench omens 192 1722 - 1600 © eee ss eee 193 
1806 - 169 6. femereneeetene 194 1897 - LOG 7 .ccetenare eee 195 
1998 - ZLO7 sce Miele Zoe a 2220 0 0s ss 0 = ore 
Ze = LOO < shutter eee 1 2364 - LoL Ge oy ae es ee 199 
22135 - 26645 sce cee cee s 200 2685 - 2676... 6 20s « scam 
2877 - DOT D «seems che 202 3096 - 3546.4 aie os 2s 
3349 = 2645 3.coeeue cs LOS 3644 = S99 0). cremeteks elements Z05 
S991 = BROT . steer ous ZOE 4408 ~- 49 15.4 eer eee Zor 
4916 - SOO 5. sae ete: s 208 jo20 — 636.1. 44 Gis ess creo 
6362 = LE ST ccetemen eens eceve 210 7440 - 89 40 oc aimee. 6 5. = 0 eee 
B94). = seelel 1 7 S eee onc 212 11174 =) 814647... - 3. Zs 
18646 =9622020 2. acre ere eee iee 22021 —= 942251. ieee Zio 
B2Z2Z52 — SGEGOSe un corer: zié 4645914 and above use .... 217 
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RSED V3.0 - Table of Adjustment Factors 
07-07-1986 

FOR DEFECT RATE ADJUSTMENT FOR DEFECT RATE ADJUSTMENT 

OVER % - THRU % FACTOR % OVER % - THRU % FACTOR % 
He 10 = 1.0..... 0.46 iC oe 0.64 
220 aHOL a ses 0.78 520 2= nu Omen oust s 2 0.90 
4.0 SIO. os OO Somme 6.0 fae se 1.09 
6.0 Wem ics faut 70) ge BO? a s08 | ez 
BO CMC 2... illic COM merO.O  ... ..2) 1.58 
=O WIPO mtasie se a een@n— iz. O foc. . 1.49 
ZO Si One a s... 1254 PimOmmemrl 4.07.04. 1.59 
a, O 1 SO are. sane 1.64 1S Om 650 ess 5 1.68 
16.0 TEs a: sees lee 2 17, Oe SSO) 2. 1.76 
18.0 TOO bi 1.80 hyo eon O 2.2.2. 1.84 
ZO. 0 2 eMO eetecoue Gil 2 Oe Oe =e. O wees 1490 
220 PAS On. tact le GS Zo Ol Mao NO Sos. 1.96 
250 BSeO  . « sais 1.99 DSO PG. Oa ees » ZO 
26.0 BO eee 2804 ZOOS. O fs.ce 2.06 
28.0 ZO Oe. 2.08 Po ROM SO. 0 soe... 2.10 
20,0 Sil OMT Sec. Zializ SOMME Oe 2.14 
mo. O RO ss «soa 1G OMENS GO) ss ss a 2s 17 
2 O 3S Oneness 2 AG SOMERS OO sins s 2. 20 
36.0 SOM aims 2.22 Om. O ... pees 
38.0 Bio. OMe oss 2. 24 Seno meson O ..... 2.25 
40.0 |. 0a 2.26 AO a O we tse 2. 26 
82.0 Gee Ome es sc 2s 27 AS Om —= 44.0 .445s 2.26 
44.0 ASP mye tases le 26 AS Oma G HO Gs6<, 2-29 
46.0 A Oe tes 2.29 ATOM AG. O <2 see Zao 
48.0 Ammen. 65 2629 AO OME ESO Osos 6. 2627 


Notes: 


WHERE SOME DEFICIENCIES ARE NOT REWORKED WITHIN ALLOTIED TIME 


i 
as 


a. Price for specified services $5,000. 00 
b. Population of services this period being billed 1,000 
Ge “FIVcespen senvwice 5.00 
d. Number of services sampled 100 
e. Number of sampled services rejected 5 
f. Observed defect (reject) rate [(e/d)x100] 5.00% 
g. Adjustment factor Note 1 ; 1.00% 
h. Deductible defect rate (£ - g) Note 2 4.00% 
i. Extrapolated defects [Whole number portion of (h x b)/100] 40 
j.- Defects observed outside the sample Note 3 23 
k. Acceptable rework completed 3 
1. Net number of services to deduct (i - k) Note 2 Si 
m. Deduct for unsatisfactory services (c x 1) $ 185.00 
n. Liquidated damages (10% of ¢ x k) Note 4 ; 1.50 
o. Other adjustments (- for deduct) Note 5S 0.00 
p. Total Payment (a -m-n +0) $4,813.50 


Adjustment factor varies with observed defect rate. 


Cannot be less than zero (0) 


Defects observed outside the sample will not be used as 
a basis for extrapolation but shall be considered in 
calculation of payment for satisfactory rework. 


Liquidated damages calculated as appropriate in accordance 
with contract provisions. 


This could include a deduction for work performed by Government 
or other forces or payment for rework from a prior billing period 
provided it would not have resulted in less than zero Net 


Services to deduct (item 1 above) in that period. 
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